Target identification method, computer equipment, device, storage medium and program product

Abstract

The invention relates to a target identification method, computer equipment, a device, a storage medium and a program product, the method is executed by a driving assistance system of a vehicle, and the method comprises the following steps: acquiring audio perception information of the vehicle; determining a first confidence degree that a sound source target belongs to a specified category based on the audio perception information; under the condition that the first confidence coefficient is greater than a first threshold value, acquiring visual perception information and radar perception information of the vehicle; under the condition that the information of the sound source target indicated by the visual perception information is not matched with the information of the sound source target indicated by the radar perception information, determining a second confidence degree that the sound source target belongs to a specified category based on the visual perception information, and determining a third confidence degree that the sound source target belongs to the specified category based on the radar perception information; performing weighted average on the first confidence coefficient, the second confidence coefficient and the third confidence coefficient to obtain a first target confidence coefficient; and determining whether the sound source target belongs to a specified category according to the first target confidence.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of assisted driving, and in particular, to a target identification method, a computer device, an apparatus, a storage medium, and a program product. BACKGROUND

[0002] With the rapid development of new energy vehicles, assisted driving functions are also increasingly valued. A driving assistance system can assist a vehicle to drive more efficiently.

[0003] In related technologies, a driving assistance system obtains radar information and image information through a radar and a camera on a vehicle, identifies whether a sound source target around the vehicle belongs to a specified category according to the radar information and the image information, and controls the vehicle to perform a corresponding operation.

[0004] However, the driving assistance system in the above scheme outputs information that is relatively single, which affects the accuracy of target identification of the driving assistance system. SUMMARY

[0005] Embodiments of the present application provide a target identification method, a computer device, an apparatus, a storage medium, and a program product, which can improve the accuracy of target identification. The technical solution is as follows:

[0006] On the one hand, a target identification method is provided, which is executed by a driving assistance system of a vehicle, and the method comprises:

[0007] obtaining audio perception information of the vehicle, the audio perception information comprising first coordinate information, first speed information, and first heading angle information of a sound source target around the vehicle;

[0008] determining a first confidence that the sound source target belongs to a specified category based on the audio perception information;

[0009] in a case where the first confidence is greater than a first threshold, obtaining visual perception information and radar perception information of the vehicle, the visual perception information comprising second coordinate information, second speed information, and second heading angle information of the sound source target, and the radar perception information comprising third coordinate information, third speed information, and third heading angle information of the sound source target;

[0010] in a case where information of the sound source target indicated by the visual perception information does not match information of the sound source target indicated by the radar perception information, determining a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determining a third confidence that the sound source target belongs to the specified category based on the radar perception information;

[0011] perform weighted average on the first confidence, the second confidence and the third confidence, and obtain a result of the weighted average as a first target confidence;

[0012] determine whether the sound source target belongs to the specified category according to the first target confidence.

[0013] In another aspect, a target recognition apparatus is provided, and the apparatus comprises:

[0014] a first obtaining module configured to obtain audio perception information of the vehicle, the audio perception information comprising first coordinate information, first speed information and first heading angle information of a sound source target around the vehicle;

[0015] a first confidence determining module configured to determine a first confidence that the sound source target belongs to a specified category based on the audio perception information;

[0016] a second obtaining module configured to, in a case where the first confidence is greater than a first threshold, obtain visual perception information and radar perception information of the vehicle, the visual perception information comprising second coordinate information, second speed information and second heading angle information of the sound source target, and the radar perception information comprising third coordinate information, third speed information and third heading angle information of the sound source target;

[0017] a second confidence determining module configured to, in a case where information of the sound source target indicated by the visual perception information does not match information of the sound source target indicated by the radar perception information, determine a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determine a third confidence that the sound source target belongs to the specified category based on the radar perception information;

[0018] a weighted average performing module configured to perform weighted average on the first confidence, the second confidence and the third confidence, and obtain a result of the weighted average as a first target confidence;

[0019] a sound source target determining module configured to determine whether the sound source target belongs to the specified category according to the first target confidence.

[0020] In a possible implementation, the case where the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information comprises the following cases:

[0021] a straight-line distance between the second coordinate information and the third coordinate information is greater than a distance threshold; and / or,

[0022] a speed difference between the second speed information and the third speed information is greater than a speed threshold; and / or,

[0023] a heading angle difference between the second heading angle information and the third heading angle information is greater than an angle threshold.

[0024] In a possible implementation, the apparatus further includes:

[0025] a fourth confidence obtaining module configured to obtain, as a fourth confidence, a confidence with a higher value from among the second confidence and the third confidence;

[0026] a second weighted average performing module configured to perform weighted average on the fourth confidence and the first confidence, and obtain a result of the weighted average as the second target confidence;

[0027] a second sound source target determining module configured to determine, according to the second target confidence, whether the sound source target belongs to the specified category.

[0028] In a possible implementation, the apparatus further includes:

[0029] a target coordinate information obtaining module configured to perform weighted average on the first coordinate information, the second coordinate information, and the third coordinate information, and obtain a result of the weighted average as target coordinate information;

[0030] a target speed information obtaining module configured to perform weighted average on the first speed information, the second speed information, and the third speed information, and obtain a result of the weighted average as target speed information;

[0031] a target heading angle information obtaining module configured to perform weighted average on the first heading angle information, the second heading angle information, and the third heading angle information, and obtain a result of the weighted average as target heading angle information;

[0032] a vehicle control module configured to control the vehicle to perform a specified operation according to the target coordinate information, the target speed information, and the target heading angle information.

[0033] In a possible implementation, the vehicle control module is configured to,

[0034] predict a motion trajectory of the sound source target based on the target coordinate information, the target speed information, and the target heading angle information;

[0035] obtain coordinate information of the vehicle, speed information of the vehicle, and heading angle information of the vehicle;

[0036] predict a collision probability of the vehicle and the sound source target according to the motion trajectory, coordinate information of the vehicle, speed information of the vehicle, and heading angle information of the vehicle;

[0037] in a case where the predicted collision probability is greater than a specified probability threshold, control the vehicle to perform an obstacle avoidance operation or a deceleration operation.

[0038] In a possible implementation, the first confidence determination module is configured to,

[0039] obtain a first coordinate confidence corresponding to the first coordinate information;

[0040] obtain a first speed confidence corresponding to the first speed information;

[0041] obtain a first heading angle confidence corresponding to the first heading angle information;

[0042] determine the first confidence corresponding to the audio perception information according to the first coordinate confidence, the first speed confidence, and the first heading angle confidence by using a softmax probability calculation method.

[0043] In another aspect, a computer device is provided, which includes a processor and a memory, the memory storing at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by the processor to implement the target identification method as described above.

[0044] In another aspect, a computer readable storage medium is provided, which stores at least one instruction, at least one program, a code set, or an instruction set, the at least one instruction, the at least one program, the code set, or the instruction set being loaded and executed by a processor to implement the target identification method as described above.

[0045] In another aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium. A processor of a computer device reads the computer program from the computer readable storage medium, and the processor executes the computer program, so that the computer device performs the target identification method provided in the various optional implementation manners described above.

[0046] The technical solution provided in the present application can include the following beneficial effects:

[0047] By introducing audio perception information in the driving assistance system, preliminary identification and confidence evaluation of the sound source target around the vehicle are realized. The first confidence is obtained based on the audio perception information, and when the first confidence is higher than a first threshold, the visual perception information and the radar perception information are further fused, and the information of different sensors is weighted, so that the driving assistance system can refer to more comprehensive perception data when performing the target task, effectively combine the actual driving condition of the vehicle, identify whether the sound source target belongs to the specified category, and thus improve the accuracy of target identification of the driving assistance system as a whole.

[0048] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory and are not restrictive of the application. BRIEF DESCRIPTION OF DRAWINGS

[0049] The accompanying drawings incorporated in and forming a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the application.

[0050] Figure 1 is a system configuration diagram of a target identification method related to an embodiment of the present application;

[0051] Figure 2 is a flowchart of a target identification method provided by an embodiment of the present application;

[0052] Figure 3 is a flowchart of a target identification method provided by an embodiment of the present application;

[0053] Figure 4 is a structure diagram of an automatic driving system architecture design provided by an embodiment of the present application;

[0054] Figure 5 is a block diagram of a target identification device provided by an exemplary embodiment of the present application;

[0055] Figure 6 is a structure diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0056] The exemplary embodiments will be described in detail herein with reference to the attached drawings. When the description below refers to the drawings, the same numbers in different drawings refer to the same or similar elements unless otherwise described. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0057] Figure 1 is a system configuration diagram of a target identification method related to an embodiment of the present application. As shown in Figure 1As shown, the vehicle 100 is controlled by the driving assistance system 10a, the driving assistance system 10a can obtain audio perception information of the vehicle, the audio perception information includes first coordinate information, first speed information and first heading angle information of a sound source target around the vehicle; determine a first confidence that the sound source target belongs to a specified category based on the audio perception information; in the case that the first confidence is greater than a first threshold, obtain visual perception information and radar perception information of the vehicle, the visual perception information includes second coordinate information, second speed information and second heading angle information of the sound source target, and the radar perception information includes third coordinate information, third speed information and third heading angle information of the sound source target; in the case that the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information, determine a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determine a third confidence that the sound source target belongs to the specified category based on the radar perception information; perform weighted average on the first confidence, the second confidence and the third confidence, and obtain a result of the weighted average as a first target confidence; determine whether the sound source target belongs to the specified category according to the first target confidence.

[0058] Figure 2 is a flowchart of a target identification method provided by an embodiment of the present application, the target identification method can include steps 210, 220, 230, 240, 250 and 260, the method can be executed by a driving assistance system of a vehicle, the vehicle can be Figure 1 vehicle 100 in the above, the driving assistance system can be Figure 1 driving assistance system 10a in the above, the implementation process of the method is as follows.

[0059] Step 210: Obtain audio perception information of the vehicle, the audio perception information includes first coordinate information, first speed information and first heading angle information of a sound source target around the vehicle.

[0060] Among them, the above-mentioned audio perception information refers to the sound information of the environment around the vehicle, in some embodiments, the above-mentioned audio perception information can be obtained by a microphone array on the vehicle.

[0061] Among them, the above-mentioned sound source target refers to an entity or phenomenon in the environment around the vehicle that can generate a sound signal that can be detected and analyzed by an audio sensing system, such as a moving object (for example, other vehicles, pedestrians), fixed facilities (for example, traffic lights), natural phenomena (such as thunder, wind).

[0062] Among them, the first coordinate information of the above-mentioned sound source target is the position of the sound source target in three-dimensional space, in some embodiments, the driving assistance system can determine the first coordinate information of the sound source by sound source positioning technology.

[0063] The first speed information of the sound source target refers to the moving speed of the sound source target. In some embodiments, the driving assistance system can estimate the speed of the sound source movement according to the rate of change of the audio signal of the sound source target.

[0064] The first heading angle information of the sound source target refers to the direction angle of the sound source target relative to the host vehicle. In some embodiments, the driving assistance system can determine the first heading angle information of the sound source target through a path tracking and control algorithm.

[0065] Step 220: Determine the first confidence that the sound source target belongs to a specified category based on the audio perception information.

[0066] The first confidence refers to the probability value that the driving assistance system determines that the sound source target belongs to a specified category (such as a police car, an ambulance, a motorcycle, or a pedestrian distress sound) based on the audio perception information.

[0067] In some embodiments, the driving assistance system determines the first confidence that the sound source target belongs to a specified category based on the audio perception information through a spectrum feature extraction and classification model. The sound source target belonging to a specified category means that the category of the sound source target belongs to the specified category.

[0068] In this example, the driving assistance system pre-processes the audio signal corresponding to the collected audio perception information, extracts the spectrum features of the audio using the Mel frequency cepstral coefficient, and inputs the extracted features into a classification model to output the first confidence that the sound source target belongs to a specified category.

[0069] In some embodiments, the driving assistance system determines the first confidence that the sound source target belongs to a specified category based on the audio perception information through a template matching and correlation analysis method.

[0070] In this example, the driving assistance system internally constructs a standard audio template library containing sound source targets of various categories. The driving assistance system performs cross-correlation analysis on the audio perception information and the template, determines whether the current audio perception information matches a template of a certain category according to the size of the correlation coefficient, and uses the correlation coefficient as the first confidence.

[0071] Step 230: When the first confidence is greater than a first threshold, obtain the visual perception information and the radar perception information of the vehicle. The visual perception information includes the second coordinate information, the second speed information, and the second heading angle information of the sound source target. The radar perception information includes the third coordinate information, the third speed information, and the third heading angle information of the sound source target.

[0072] The first threshold is a preset confidence threshold (for example, 0.7), which is used to determine whether the driving assistance system needs to call other sensors for further verification.

[0073] The visual perception information is information extracted by the driving assistance system from images or videos collected by visual sensors around the vehicle. For example, the visual perception information can be information obtained by a camera of the vehicle through image recognition algorithms.

[0074] The radar perception information refers to physical state information of a target object extracted by the driving assistance system through radar sensors of the vehicle, such as millimeter wave radar or laser radar, after detecting the surrounding environment. The physical state information can be coordinate information, speed information, and heading angle information of the sound source target.

[0075] Step 240: In the case where the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information, determining a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determining a third confidence that the sound source target belongs to the specified category based on the radar perception information.

[0076] The information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information means that there is a significant difference between the visual perception information and the radar perception information.

[0077] In some embodiments, the driving assistance system can input the visual perception information into an image classification model to obtain the second confidence output by the image classification model.

[0078] In some embodiments, the driving assistance system can determine the third confidence that the sound source target belongs to the specified category according to the radar perception information through point cloud feature matching technology or target contour analysis technology.

[0079] Step 250: Perform weighted averaging on the first confidence, the second confidence, and the third confidence, and obtain a result of the weighted averaging as the first target confidence.

[0080] The weighted averaging means that different weights are respectively assigned to the first confidence, the second confidence, and the third confidence for fusion calculation. For example, the weights can be set as: the first confidence 0.3, the second confidence 0.4, and the third confidence 0.3.

[0081] In other embodiments, the driving assistance system adjusts the weights assigned to the first confidence, the second confidence, and the third confidence according to the environment in which the vehicle is located. The environment in which the vehicle is located is at least one of a road environment and a climate condition.

[0082] Optionally, the driving assistance system adjusts the weights of the first confidence, the second confidence and the third confidence according to an environment in which the vehicle is located, the environment being at least one of a road environment and a weather condition, and the environment information table including a corresponding relationship between the environment in which the vehicle is located and the weights of the first confidence, the second confidence and the third confidence, that is, in the environment information table, an environment corresponds to the weights of the first confidence, the second confidence and the third confidence, and different environments correspond to different weights of the first confidence, the second confidence and the third confidence.

[0083] For example, in foggy weather, the quality of visual perception information decreases, and the driving assistance system can reduce the weight of the second confidence and increase the weight of radar perception information, that is, the third confidence.

[0084] In the above embodiment, the driving assistance system dynamically adjusts the weights of different types of perception data according to the environment in which the vehicle is located, so that the driving assistance system focuses on different perception information when performing a weighted average operation, and can combine the actual driving scene of the vehicle, thereby effectively improving the weighted average result and improving the accuracy of the first target confidence.

[0085] Step 260: determining whether the sound source target belongs to a specified category according to the first target confidence.

[0086] The first target confidence is a final recognition probability obtained by integrating audio, visual and radar perception information, and is a comprehensive probability value of whether the sound source target belongs to the specified category, which is calculated by fusing audio, visual and radar perception information.

[0087] In some embodiments, the driving assistance system compares the first target confidence with a decision threshold, and in a case where the first target confidence is greater than the decision threshold, it is determined that the sound source target belongs to the specified category, and in a case where the first target confidence is less than the decision threshold, it is determined that the sound source target does not belong to the specified category.

[0088] The decision threshold is a preset ratio, for example, 0.8.

[0089] In the embodiments of the present application, by introducing audio perception information into the driving assistance system, preliminary identification and confidence evaluation of the sound source target around the vehicle are realized, the first confidence is obtained based on the audio perception information, and when the first confidence is higher than the first threshold, the visual perception information and the radar perception information are further fused, the information of different sensors is weighted, so that the driving assistance system can refer to more comprehensive perception data when performing a target task, effectively combine the actual driving situation of the vehicle, and determine whether the sound source target belongs to the specified category, thereby improving the accuracy of target recognition of the driving assistance system as a whole.

[0090] Based on the scheme shown in any of the above embodiments, in a possible implementation, the case that the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information includes the following cases:

[0091] a straight-line distance between the second coordinate information and the third coordinate information is greater than a distance threshold; and / or,

[0092] a speed difference between the second speed information and the third speed information is greater than a speed threshold; and / or,

[0093] a heading angle difference between the second heading angle information and the third heading angle information is greater than an angle threshold.

[0094] In some embodiments, the driving assistance system can calculate the Euclidean distance between two coordinate points, obtain the straight-line distance between the second coordinate information and the third coordinate information, and if the straight-line distance between the second coordinate information and the third coordinate information exceeds the distance threshold (such as 0.5 meters), it indicates that the error between the second coordinate information and the third coordinate information is large, and the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information.

[0095] In some embodiments, after obtaining the second speed information and the third speed information, the driving assistance system calculates the speed difference between the second speed information and the third speed information: |v1-v2|, and if the speed difference between the second speed information and the third speed information exceeds the speed threshold (such as 1.5 m / s), it indicates that the moving speed difference of the sound source target indicated by the second speed information and the third speed information is large, and the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information.

[0096] In some embodiments, after obtaining the second heading angle information and the third heading angle information, the driving assistance system calculates the heading angle difference between the second heading angle information and the third heading angle information, and in the case that the heading angle difference between the second heading angle information and the third heading angle information is greater than the angle threshold (such as 20°), it indicates that the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information.

[0097] In the embodiments of the present application, the driving assistance system sets the difference threshold between the coordinates, speeds, and heading angles, and clearly defines the specific judgment standard for the case that the visual perception information does not match the radar perception information, thereby improving the reliability of the driving assistance system in judging the case that the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information.

[0098] Based on the solutions shown in any one or more of the above embodiments, in one possible implementation, step 220 can be implemented as follows: the driving assistance system obtains the first coordinate confidence level corresponding to the first coordinate information; obtains the first speed confidence level corresponding to the first speed information; obtains the first heading angle confidence level corresponding to the first heading angle information; and determines the first confidence level corresponding to the audio perception information by using the softmax probability calculation method based on the first coordinate confidence level, the first speed confidence level, and the first heading angle confidence level.

[0099] The aforementioned first coordinate confidence level refers to the probability value by which the assisted driving system determines that the sound source target belongs to a specified category (such as police car, ambulance, motorcycle, or pedestrian calling for help) based on the first coordinate information. The first coordinate confidence level is a value between 0 and 1.

[0100] The aforementioned first speed confidence refers to the probability value by which the driver assistance system determines that the sound source target belongs to a specified category (such as police car, ambulance, motorcycle, or pedestrian calling for help) based on the first speed information, and the first coordinate confidence is a value between 0 and 1.

[0101] The aforementioned first heading angle confidence level refers to the probability value by which the assisted driving system determines that the sound source target belongs to a specified category (such as police car, ambulance, motorcycle, or pedestrian calling for help) based on the first heading angle information. The first coordinate confidence level is a value between 0 and 1.

[0102] In some embodiments, the driving assistance system can use the confidence scores of the above three dimensions (first coordinate confidence score, first speed confidence score, and first heading angle confidence score) as input features, and then use the softmax function to weighted fuse them to obtain a comprehensive category confidence score as the first confidence score corresponding to the audio perception information.

[0103] For example, the confidence scores for the first coordinate, first speed, and first heading angle are 0.8, 0.7, and 0.75, respectively. The driver assistance system calculates and outputs a comprehensive confidence score of 0.82 using the softmax function, which is used as the first confidence score corresponding to the audio perception information.

[0104] In this embodiment, the driving assistance system obtains a comprehensive first confidence level by using the softmax probability calculation method to calculate the confidence levels corresponding to the coordinates, speed, and heading angle in the audio perception information. This achieves weighted fusion of information from various dimensions within the audio perception information, normalizes the multi-dimensional information, improves the scientificity and rationality of the calculation of the first confidence level, and enhances the accuracy of the driving assistance system in identifying the target category of the sound source.

[0105] based on Figure 2 For the target recognition method shown, please refer to...Figure 3 , Figure 3 is a flowchart of a target identification method provided by an embodiment of the present application, and the target identification method further includes steps 270, 280 and 290, which are as follows.

[0106] Step 270: obtaining the confidence degree with a higher value between the second confidence degree and the third confidence degree as the fourth confidence degree.

[0107] The second confidence degree refers to the probability that the driving assistance system determines the sound source target to belong to the specified category based on the visual perception information.

[0108] The third confidence degree refers to the probability that the driving assistance system determines the sound source target to belong to the specified category based on the radar perception information.

[0109] The fourth confidence degree is the one with a higher value between the second confidence degree and the third confidence degree, representing the one that is more reliable in identifying the target category in the visual perception information and the radar perception information.

[0110] Step 280: performing weighted average on the fourth confidence degree and the first confidence degree to obtain a result of the weighted average as a second target confidence degree.

[0111] In some embodiments, the driving assistance system performs weighted average on the fourth confidence degree and the first confidence degree according to a preset assigned weight to obtain a result of the weighted average as the second target confidence degree.

[0112] Step 290: determining whether the sound source target belongs to the specified category according to the second target confidence degree.

[0113] In some embodiments, the driving assistance system sets a determination threshold, compares the second target confidence degree with the determination threshold, and determines whether the sound source target belongs to the specified category according to a comparison result. If the second target confidence degree is greater than the determination threshold, it is determined that the sound source target belongs to the specified category.

[0114] In the embodiments of the present application, the driving assistance system performs weighted average on the fourth confidence degree with a higher confidence degree between the visual perception information and the radar perception information and the first confidence degree of the audio perception information, further optimizes the multi-modal information fusion strategy, and improves the accuracy of target identification.

[0115] Based on the above-mentioned scheme corresponding to one or more embodiments, in a possible implementation, the driving assistance system of the vehicle performs weighted averaging on the first coordinate information, the second coordinate information, and the third coordinate information, obtains a result of the weighted averaging as target coordinate information; performs weighted averaging on the first speed information, the second speed information, and the third speed information, obtains a result of the weighted averaging as target speed information; performs weighted averaging on the first heading angle information, the second heading angle information, and the third heading angle information, obtains a result of the weighted averaging as target heading angle information; and controls the vehicle to perform a specified operation according to the target coordinate information, the target speed information, and the target heading angle information.

[0116] The target coordinate information is more accurate target coordinate information obtained by weighted fusion of the first coordinate information, the second coordinate information, and the third coordinate information.

[0117] The target speed information is more accurate target speed information obtained by weighted fusion of the first speed information, the second speed information, and the third speed information.

[0118] The target heading angle information is more accurate target heading angle information obtained by weighted fusion of the first heading angle information, the second heading angle information, and the third heading angle information.

[0119] The specified operation includes but is not limited to deceleration, lane changing, obstacle avoidance, emergency braking, and other driving assistance or automatic driving behaviors.

[0120] In some embodiments, the driving assistance system performs trajectory prediction on the sound source target according to the target coordinate information, the target speed information, and the target heading angle information; and controls the vehicle to perform a specified operation based on a result of the trajectory prediction of the sound source target.

[0121] The trajectory prediction is based on coordinate information, speed information, and heading angle information of the sound source target to predict a motion trajectory of the sound source target within a specified time period in the future.

[0122] In some embodiments, the driving assistance system can input the target coordinate information, the target speed information, and the target heading angle information into a trajectory prediction model, obtain a prediction result output by the trajectory prediction model, and the prediction result indicates a motion trajectory of the sound source target within a specified time period in the future.

[0123] In this embodiment, after determining the future trajectory of the sound source target, the driving assistance system combines the path planning of the vehicle itself to determine the operation to be performed by the vehicle, for example, the trajectory prediction result of the sound source target indicates that the sound source target will reach coordinate point A after 1 minute in the future, and the vehicle itself will also reach point A after 1 minute in the future, at this time, the vehicle is controlled to perform a deceleration or acceleration operation to reach point A in advance or delay, to avoid collision with the sound source target.

[0124] In the embodiments of the present application, the driving assistance system obtains more accurate state estimation information of the sound source target by performing weighted average on the coordinate information, speed information and heading angle information in the audio perception information, visual perception information and radar perception information, and controls the vehicle to perform a specified operation according to the state estimation information, which can effectively improve the control effect of the vehicle.

[0125] Based on the above-mentioned scheme corresponding to one or more embodiments, in a possible implementation, the driving assistance system of the vehicle predicts the motion trajectory of the sound source target based on the target coordinate information, the target speed information and the target heading angle information; obtains the coordinate information of the vehicle, the speed information of the vehicle and the heading angle information of the vehicle; predicts the collision probability of the vehicle and the sound source target according to the motion trajectory, the coordinate information of the vehicle, the speed information of the vehicle and the heading angle information of the vehicle; and controls the vehicle to perform an obstacle avoidance operation or a deceleration operation in the case that the predicted collision probability is greater than a specified probability threshold.

[0126] In some embodiments, the driving assistance system of the vehicle can input the target coordinate information, the target speed information and the target heading angle information into a motion trajectory prediction model, and obtain the motion trajectory of the sound source target output by the motion trajectory prediction model.

[0127] In some embodiments, the driving assistance system performs trajectory prediction of the sound source target according to the target coordinate information, the target speed information and the target heading angle information, and controls the vehicle to perform a specified operation based on the result of the trajectory prediction of the sound source target, and the prediction operation includes trajectory prediction.

[0128] The above-mentioned coordinate information of the vehicle indicates the position of the vehicle in space, and in some embodiments, the coordinate information can be obtained by an inertial measurement unit of the vehicle.

[0129] The above-mentioned speed information of the vehicle indicates the current speed of the vehicle, and in some embodiments, the speed information can be obtained by a speed sensor.

[0130] The above-mentioned heading angle information of the vehicle indicates the driving direction of the vehicle, i.e. the included angle between the current motion direction of the vehicle and the north direction, and in some embodiments, the heading angle information can be obtained by a GPS (Global Positioning System).

[0131] wherein the above collision probability prediction is predicting a probability of a collision between the sound source target and the vehicle in a future specified time period, in combination with a current state of the vehicle.

[0132] In some embodiments, the driving assistance system inputs the target coordinate information, the target speed information, and the target heading angle information into the collision prediction model, and obtains a prediction result output by the collision prediction model, the prediction result indicating a probability of a collision between the sound source target and the vehicle in a future specified time period. In this embodiment, the collision probability is a probability value between 0 and 1, and the greater the probability value, the greater the possibility of a collision between the sound source target and the vehicle in the future specified time period.

[0133] wherein the above specified probability threshold is a preset safety threshold (for example, 0.7), used by the driving assistance system to determine whether to take measures on the vehicle, to perform an obstacle avoidance operation or a speed reduction operation.

[0134] wherein the above obstacle avoidance operation is to control the vehicle to change the driving direction to avoid the sound source target.

[0135] wherein the above speed reduction operation is to reduce the vehicle speed to increase the safety distance between the vehicle and the sound source target or to avoid a collision.

[0136] In the embodiments of the present application, after the driving assistance system completes the collision probability prediction, the predicted collision probability is obtained, and the predicted collision probability is compared with the specified probability threshold. In the case that the predicted collision probability is greater than the specified probability threshold, the vehicle is controlled to perform the obstacle avoidance operation or the speed reduction operation.

[0137] In the embodiments of the present application, the driving assistance system predicts the motion trajectory of the sound source target based on the fused target state information, and predicts the collision probability in combination with the state of the vehicle itself, so as to realize dynamic evaluation of potential risks, perform the obstacle avoidance or speed reduction operation when the predicted collision probability exceeds the specified probability threshold, and effectively improve the driving safety, thereby improving the vehicle control effect as a whole.

[0138] For example, based on the above-mentioned collision probability prediction, the driving assistance system can perform the obstacle avoidance operation or the speed reduction operation when the predicted collision probability exceeds the specified probability threshold. Figure 2 to Figure 3 According to any one or more embodiments, the present application provides a method for designing an autonomous driving system architecture that fuses audio sensor input.

[0139] The embodiments of the present application provide an auxiliary driving / autonomous driving system that adds audio sensor input on the basis of traditional vision / radar sensors, which is referred to as an ADAS (Advanced Driver Assistance Systems, high-level driving assistance system) system architecture design method.

[0140] Existing ADAS systems generally use visual perception sensors, millimeter wave radars and lidars as inputs of the ADAS system. It is similar to the vision of human drivers from the perspective of bionics. The ADAS system will react to the traffic information such as obstacles and lane lines that the sensor "sees".

[0141] Problems of the existing solutions: the existing sensors have certain performance defects in some special scenarios, which cannot perceive the target or obstacle, or the confidence of the perception result itself cannot reach 100%. The above situations will increase the number of MPI (Manual Pilot Intervention) takeovers, and if the human driver does not take over in time, it may even cause safety risks. Even the VLM (Vision-Language Model) that is being applied more and more is based on the perception of a pure visual large model, which has a certain common cause failure risk from the perspective of functional safety.

[0142] The architecture design method improves the perception ability of surrounding traffic participants by adding audio sensor inputs, so that the ADAS system can drive the vehicle in a more human-like way (eyes can see six directions and ears can hear eight directions).

[0143] The working principle of the above architecture system is to add audio sensors (such as microphones) around the vehicle body, and analyze and process the digital audio signals through DSP (Digital Signal Processor) or AI (Artificial Intelligence) models.

[0144] When signals such as other vehicles honking or hitting, explosions, etc. are identified, appropriate measures are taken, such as alarming, slowing down, turning, etc.

[0145] Please refer to Figure 4 , Figure 4 is a structural diagram of an automatic driving system architecture design provided by an embodiment of the present application.

[0146] As shown in Figure 4 , the architecture design is to add an audio perception module in front of the traditional ADAS system architecture, and the audio perception module includes an audio sensor, an audio signal processing module and an audio perception module.

[0147] In the sensor fusion module, the fusion of the dynamic target of the audio perception output (the dynamic target refers to a moving vehicle, a pedestrian or other traffic participants) needs to be increased. The fusion is a technical process of comprehensively processing and analyzing the target data and information perceived by different sensors. If there is a conflict (the conflict refers to the deviation of data such as coordinates, types and speeds of the same target perceived by different sensors), the priority is determined according to the confidence of the respective perception results (information such as coordinates, types and speeds of the target).

[0148] In some embodiments, the classification confidence of a single sensor is based on the softmax probability calculation in deep learning, and the confidence score is the highest. The confidence calculation method after multi-sensor fusion is, for example, a simple weighted average, different weights are given according to the reliability of different sensors in a specific scene, and finally the confidence of the same type of result is weighted and averaged.

[0149] In the embodiments of the present application, the audio perception output provides the coordinates, speed and heading angle information of the dynamic target. The above-mentioned audio sensor can be realized by deploying a possible array around the whole vehicle.

[0150] In the embodiments of the present application, the driving assistance system can realize sound positioning by using, for example, a (Time Difference of Arrival, TDOA) algorithm.

[0151] Among them, the above-mentioned sound positioning (Sound Source Localization, SSL) based on a microphone array is a core task of acoustic signal processing, which aims to estimate the direction (DOA, Direction of Arrival) or accurate position (3D coordinates) of the sound source in space by using the differences in sound signals received by multiple spatially distributed microphones. Core principle: use the propagation characteristics of sound waves. There is a time difference (TDOA, Time Difference of Arrival), a phase difference and an energy attenuation difference between the arrival of sound at different microphones, and these differences contain the direction or position information of the sound source:

[0152] Time difference: sound waves propagate at a fixed speed (~ 340 m / s), and the time difference between the arrival of sound at different microphones is directly related to the path difference between the sound source and the microphone.

[0153] Phase difference: the phase difference between the signals received by different microphones is related to the frequency and TDOA.

[0154] Energy attenuation: the farther the microphone is from the sound source, the lower the energy of the received signal (following the inverse square law).

[0155] Key algorithm example: based on time delay estimation, the core idea is to estimate the time difference of arrival of sound source to different microphone pairs first, and then reverse the sound source position through geometric relationship.

[0156] The cross-correlation of two microphone signals xi(t) and xj(t) can be calculated by formula one:

[0157]

[0158] wherein, the above Φ(f) is a weighting function.

[0159] TDOA estimation: τij=argmaxτRij(τ)

[0160] Position solution: given TDOA τij, the sound source satisfies formula two hyperbolic equation:

[0161] ‖p-m_i‖-‖p-m i ‖=c·τ ij Formula two

[0162] Multiple sets of TDOA form an equation group, which can be solved for the sound source coordinates p through nonlinear optimization (such as Levenberg-Marquardt) or closed-form solution (such as SVD).

[0163] Velocity, heading angle and other information are obtained by difference of continuous multi-needle position information and time.

[0164] In the embodiments of the present application, the fusion strategy is as follows.

[0165] The processing of audio perception results can be divided into three cases:

[0166] 1) Confidence < threshold L1, audio perception result is not output to downstream (downstream modules of data flow, such as sensor fusion, path planning, vehicle control, etc.), or invalid value is output (output invalid flag, downstream determines whether to process according to the flag).

[0167] 2) L1<=confidence<L2, audio perception result directly issues audio image light alarm through alarm prompt module (path ①), without vehicle control. Figure 4

[0168] 3) Confidence >=L2, audio perception result participates in sensor fusion, and vehicle control is performed through downstream modules (path ②), such as lateral avoidance, braking, etc. Figure 4

[0169] Note: Thresholds L1 and L2 are calibratable values.

[0170] ​​In some embodiments, the automatic driving system architecture design method of fusing audio sensor input is applicable to passenger cars and commercial vehicles, and the perception range of audio is not extended from vehicle horn sound to other sound sources, such as impact, explosion, etc.

[0171] In summary, the automatic driving system architecture design method of fusing audio sensor input can improve the overall perception performance of the ADAS system in specific scenarios, further improving driving safety.

[0172] Please refer to Figure 5 , which shows a block diagram of a target identification device provided by an exemplary embodiment of the present application. The input target identification device can be implemented as all or part of a computer device by hardware or a combination of hardware and software, to implement all or part of the steps in the above Figure 2 to Figure 3 embodiments.

[0173] As Figure 5 shown, the target identification device includes:

[0174] A first acquisition module 51 is configured to acquire audio perception information of a vehicle, the audio perception information including first coordinate information, first speed information, and first heading angle information of a sound source target around the vehicle;

[0175] A first confidence determination module 52 is configured to determine a first confidence that the sound source target belongs to a specified category based on the audio perception information;

[0176] A second acquisition module 53 is configured to acquire visual perception information and radar perception information of the vehicle in a case where the first confidence is greater than a first threshold, the visual perception information including second coordinate information, second speed information, and second heading angle information of the sound source target, and the radar perception information including third coordinate information, third speed information, and third heading angle information of the sound source target;

[0177] A second confidence determination module 54 is configured to determine a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determine a third confidence that the sound source target belongs to the specified category based on the radar perception information, in a case where information of the sound source target indicated by the visual perception information does not match information of the sound source target indicated by the radar perception information;

[0178] A weighted average execution module 55 is configured to perform weighted average on the first confidence, the second confidence, and the third confidence, and obtain a result of the weighted average as a first target confidence;

[0179] A sound source target determination module 56 is configured to determine whether the sound source target belongs to the specified category according to the first target confidence.

[0180] In a possible implementation, the case that the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information includes the following cases:

[0181] a straight-line distance between the second coordinate information and the third coordinate information is greater than a distance threshold; and / or,

[0182] a speed difference between the second speed information and the third speed information is greater than a speed threshold; and / or,

[0183] a heading angle difference between the second heading angle information and the third heading angle information is greater than an angle threshold.

[0184] In a possible implementation, the apparatus further includes:

[0185] a fourth confidence obtaining module, configured to obtain, as a fourth confidence, a confidence with a higher value from among the second confidence and the third confidence;

[0186] a second weighted average performing module, configured to perform weighted average on the fourth confidence and the first confidence, and obtain a result of the weighted average as a second target confidence;

[0187] a second sound source target determining module, configured to determine, according to the second target confidence, whether the sound source target belongs to a specified category.

[0188] In a possible implementation, the apparatus further includes:

[0189] a target coordinate information obtaining module, configured to perform weighted average on the first coordinate information, the second coordinate information, and the third coordinate information, and obtain a result of the weighted average as target coordinate information;

[0190] a target speed information obtaining module, configured to perform weighted average on the first speed information, the second speed information, and the third speed information, and obtain a result of the weighted average as target speed information;

[0191] a target heading angle information obtaining module, configured to perform weighted average on the first heading angle information, the second heading angle information, and the third heading angle information, and obtain a result of the weighted average as target heading angle information;

[0192] a vehicle control module, configured to control the vehicle to perform a specified operation according to the target coordinate information, the target speed information, and the target heading angle information.

[0193] In a possible implementation, the vehicle control module is configured to,

[0194] predict a motion trajectory of the sound source target based on the target coordinate information, the target speed information, and the target heading angle information;

[0195] obtain coordinate information of the vehicle, speed information of the vehicle, and heading angle information of the vehicle;

[0196] predict a collision probability of the vehicle and the sound source target according to the motion trajectory, the coordinate information of the vehicle, the speed information of the vehicle, and the heading angle information of the vehicle;

[0197] in a case where the predicted collision probability is greater than a specified probability threshold, control the vehicle to perform an obstacle avoidance operation or a deceleration operation.

[0198] In one possible implementation, the first confidence determination module 52 is configured to,

[0199] obtain a first coordinate confidence corresponding to the first coordinate information;

[0200] obtain a first speed confidence corresponding to the first speed information;

[0201] obtain a first heading angle confidence corresponding to the first heading angle information;

[0202] determine the first confidence corresponding to the audio perception information according to the first coordinate confidence, the first speed confidence, and the first heading angle confidence by using a softmax probability calculation method.

[0203] Please refer to Figure 6 , Figure 6 is a structural schematic diagram of a computer device provided in an example embodiment of the present application. The computer device 600 includes a central processing unit (CPU) 601, a system memory 604 including a random access memory (RAM) 602 and a read-only memory (ROM) 603, and a system bus 605 connecting the system memory 604 and the central processing unit 601. The computer device 600 further includes a basic input / output system (I / O system) 606 helping to transfer information between various devices in the computer, and a mass storage device 607 for storing an operating system 613, application programs 614, and other program modules 615.

[0204] The basic input / output system 606 includes the various components that are used to display information for the user and input devices 609 such as a mouse, keyboard, etc. for entering commands and information. Both the display 608 and input devices 609 are connected to the central processing unit 601 through the input / output controller 610 which is connected to the system bus 605. The basic input / output system 606 can also include the input / output controller 610 for receiving and processing input from a number of other devices, including a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 provides output to the display screen, a printer, or other type of output device.

[0205] The mass storage device 607 is connected to the central processing unit 601 through a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer readable media provide non-volatile storage for the computer device 600. That is, the mass storage device 607 can include a computer readable medium (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0206] Without loss of generality, the computer readable media can include computer storage media and communication media. Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes RAM (Random Access Memory), ROM (Read-Only Memory), EPROM (Erasable Programmable Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), flash memory or other solid state memory technology, CD-ROM, digital video disk (DVD), or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices. The computer storage media is not limited to the foregoing examples. The system memory 604 and the mass storage device 607 are examples of computer storage media. The computer readable media also includes communication media including any medium that facilitates the transfer of a computer program from one place to another. A connection 612, where the connection 612 is an example of a communication medium, facilitates the transfer of computer program for execution between the mass storage device 607 and the RAM 604. Clearly, one of ordinary skill can appreciate that the computer readable media can be embodied in a computer program product.

[0207] The computer device 600 can be connected to the Internet or other network through a network interface unit 611 connected to the system bus 605.

[0208] The memory also includes one or more programs stored therein, which one or more programs are executable by the central processing unit 601 to implement Figure 2 to Figure 3 all or part of the steps of the methods shown in the above various embodiments of the present application.

[0209] In an example embodiment, there is also provided a chip, comprising programmable logic circuitry and / or program instructions, which when the chip is run on a computer device, is used to implement all or part of the steps of the methods shown in the above various embodiments of the present application.

[0210] In an example embodiment, there is also provided a computer program product, comprising computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor reads and executes the computer instructions from the computer readable storage medium to implement all or part of the steps of the methods shown in the above various embodiments of the present application.

[0211] In an example embodiment, there is also provided a computer readable storage medium, having stored therein a computer program, which is loaded and executed by a processor to implement all or part of the steps of the methods shown in the above various embodiments of the present application.

[0212] It is understood by those skilled in the art that all or part of the steps of the above embodiments can be completed by hardware, or by program instructions to relevant hardware, and the program can be stored in a computer readable storage medium, and the storage medium mentioned above can be a read-only memory, a disk or an optical disk, etc.

[0213] It is understood by those skilled in the art that in the above one or more examples, the functions described in the embodiments of the present application can be implemented by hardware, software, firmware or any combination thereof. When implemented by software, these functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes computer storage medium and communication medium, wherein the communication medium includes any medium that facilitates the transfer of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0214] The above is only optional embodiments of the present application, and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A target recognition method characterized by, The method is executed by a driving assistance system of a vehicle, and the method comprises: obtaining audio perception information of the vehicle, the audio perception information comprising first coordinate information, first speed information and first heading angle information of a sound source target around the vehicle; determining a first confidence that the sound source target belongs to a specified category based on the audio perception information; in a case where the first confidence is greater than a first threshold, obtaining visual perception information and radar perception information of the vehicle, the visual perception information comprising second coordinate information, second speed information and second heading angle information of the sound source target, and the radar perception information comprising third coordinate information, third speed information and third heading angle information of the sound source target; in a case where information of the sound source target indicated by the visual perception information does not match information of the sound source target indicated by the radar perception information, determining a second confidence that the sound source target belongs to the specified category based on the visual perception information, and determining a third confidence that the sound source target belongs to the specified category based on the radar perception information; performing weighted averaging on the first confidence, the second confidence and the third confidence, and obtaining a result of the weighted averaging as a first target confidence; determining whether the sound source target belongs to the specified category according to the first target confidence.

2. The method of claim 1, wherein, The case where the information of the sound source target indicated by the visual perception information does not match the information of the sound source target indicated by the radar perception information comprises: a straight-line distance between the second coordinate information and the third coordinate information is greater than a distance threshold; and / or, a speed difference between the second speed information and the third speed information is greater than a speed threshold; and / or, a heading angle difference between the second heading angle information and the third heading angle information is greater than an angle threshold.

3. The method of claim 1, wherein, The method further comprises: obtaining a confidence with a higher value between the second confidence and the third confidence as a fourth confidence; performing weighted averaging on the fourth confidence and the first confidence, and obtaining a result of the weighted averaging as a second target confidence; determining whether the sound source target belongs to the specified category according to the second target confidence.

4. The method of claim 1, wherein, The method further comprises: performing weighted averaging on the first coordinate information, the second coordinate information and the third coordinate information, and obtaining a result of the weighted averaging as target coordinate information; performing weighted averaging on the first speed information, the second speed information and the third speed information, and obtaining a result of the weighted averaging as target speed information; performing weighted averaging on the first heading angle information, the second heading angle information and the third heading angle information, and obtaining a result of the weighted averaging as target heading angle information; controlling the vehicle to perform a specified operation according to the target coordinate information, the target speed information and the target heading angle information.

5. The method of claim 4, wherein, The controlling the vehicle to perform the specified operation according to the target coordinate information, the target speed information and the target heading angle information comprises: predict a motion trajectory of the sound source target based on the target coordinate information, target speed information, and the target heading angle information; obtain coordinate information of the vehicle, speed information of the vehicle, and heading angle information of the vehicle; predict a collision probability of the vehicle and the sound source target according to the motion trajectory, the coordinate information of the vehicle, the speed information of the vehicle, and the heading angle information of the vehicle; in a case where the predicted collision probability is greater than a specified probability threshold, control the vehicle to perform an obstacle avoidance operation or a deceleration operation.

6. The method according to any one of claims 1 to 5, characterized in that, The first confidence degree that the sound source target belongs to the specified category based on the audio perception information includes: obtaining a first coordinate confidence degree corresponding to the first coordinate information; obtaining a first speed confidence degree corresponding to the first speed information; obtaining a first heading angle confidence degree corresponding to the first heading angle information; determining a first confidence degree corresponding to the audio perception information by a softmax probability calculation method according to the first coordinate confidence degree, the first speed confidence degree, and the first heading angle confidence degree.

7. A target recognition device, characterized by The device includes: a first obtaining module configured to obtain audio perception information of the vehicle, the audio perception information including first coordinate information, first speed information, and first heading angle information of a sound source target around the vehicle; a first confidence degree determination module configured to determine a first confidence degree that the sound source target belongs to a specified category based on the audio perception information; a second obtaining module configured to, in a case where the first confidence degree is greater than a first threshold, obtain visual perception information and radar perception information of the vehicle, the visual perception information including second coordinate information, second speed information, and second heading angle information of the sound source target, and the radar perception information including third coordinate information, third speed information, and third heading angle information of the sound source target; a second confidence degree determination module configured to, in a case where information of the sound source target indicated by the visual perception information does not match information of the sound source target indicated by the radar perception information, determine a second confidence degree that the sound source target belongs to the specified category based on the visual perception information, and determine a third confidence degree that the sound source target belongs to the specified category based on the radar perception information; a weighted average execution module configured to perform weighted average on the first confidence degree, the second confidence degree, and the third confidence degree, and obtain a result of the weighted average as a first target confidence degree; a sound source target determination module configured to determine whether the sound source target belongs to the specified category according to the first target confidence degree.

8. A computer device, comprising: The computer device includes a processor and a memory, and the memory stores instructions executed by the processor to implement the target identification method of any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores instructions executed by the processor of the computer device to implement the target identification method of any one of claims 1 to 6.

10. A computer program product, characterised in that, The computer program product comprises computer instructions stored in a computer readable storage medium; the computer instructions are read and executed by a processor of a computer device to implement the target identification method according to any one of claims 1 to 6.

Images