In-vehicle target positioning method, device, equipment and medium

Abstract

The invention provides an in-vehicle target positioning method and device, equipment and a medium. The in-vehicle target positioning method comprises the steps that in-vehicle visual data and in-vehicle sound data are acquired; performing time-space synchronization on the in-vehicle visual data and the in-vehicle sound data; positioning an in-vehicle target according to the in-vehicle visual data after time-space synchronization to obtain a visual positioning result of the in-vehicle target; positioning an in-vehicle target according to the in-vehicle sound data after time-space synchronization to obtain an acoustic positioning result of the in-vehicle target; and determining in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result. According to the invention, the accuracy of in-vehicle target positioning can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular, to a method, device, equipment and medium for positioning targets inside a vehicle. Background Art

[0002] With the intelligence of vehicles, the positioning of people inside the vehicle can be achieved through artificial intelligence technology.

[0003] In related technologies, the sound of people inside the vehicle is collected through a microphone array deployed inside the vehicle; based on the sound of people inside the vehicle, sound source localization is performed to obtain the position of people inside the vehicle.

[0004] However, the above method has low positioning accuracy for targets inside the vehicle. Summary of the Invention

[0005] Based on the above technical status quo, this application provides a method, device, equipment and medium for positioning targets inside a vehicle, which can improve the positioning accuracy of targets inside the vehicle.

[0006] To achieve the above technical objectives, the following technical solutions are specifically proposed in this application:

[0007] According to the first aspect of the embodiments of this application, a method for positioning targets inside a vehicle is provided, including: obtaining visual data inside the vehicle and sound data inside the vehicle; performing spatio-temporal synchronization on the visual data inside the vehicle and the sound data inside the vehicle; performing positioning of the target inside the vehicle according to the spatio-temporally synchronized visual data inside the vehicle to obtain a visual positioning result of the target inside the vehicle; performing positioning of the target inside the vehicle according to the spatio-temporally synchronized sound data inside the vehicle to obtain an acoustic positioning result of the target inside the vehicle; determining the position information inside the vehicle of the target inside the vehicle according to the visual positioning result and the acoustic positioning result.

[0008] In some implementation manners, in the spatio-temporal synchronization of the visual data inside the vehicle and the sound data inside the vehicle, the time synchronization process of the visual data inside the vehicle and the sound data inside the vehicle includes: establishing a unified time stamp for the visual data inside the vehicle and the sound data inside the vehicle to obtain the visually synchronized visual data inside the vehicle and the sound data inside the vehicle after time synchronization.

[0009] In some implementation manners, in the spatio-temporal synchronization of the visual data inside the vehicle and the sound data inside the vehicle, the space synchronization process of the visual data inside the vehicle and the sound data inside the vehicle includes: converting the visually synchronized visual data inside the vehicle from the visual sensor coordinate system to a set coordinate system inside the vehicle to obtain the spatio-temporally synchronized visual data inside the vehicle; converting the sound data inside the vehicle after time synchronization from the acoustic sensor coordinate system to the coordinate system inside the vehicle to obtain the spatio-temporally synchronized sound data inside the vehicle.

[0010] In some implementations, the positioning of the in-vehicle target based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual positioning result of the in-vehicle target includes: performing target detection on the spatio-temporally synchronized in-vehicle visual data to obtain the visual positioning coordinates of the in-vehicle target and the visual detection score corresponding to the visual positioning coordinates; determining the confidence score corresponding to the visual positioning coordinates according to the optical radiation index of the in-vehicle visual data and the visual detection score; wherein the visual positioning result includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates.

[0011] In some implementations, the determining the confidence score corresponding to the visual positioning coordinates according to the optical radiation index of the in-vehicle visual data and the visual detection score includes: weighting the optical radiation index and the visual detection score according to the weight parameter corresponding to the optical radiation index and the weight parameter corresponding to the visual detection score to obtain the confidence score corresponding to the visual positioning coordinates.

[0012] In some implementations, the positioning of the in-vehicle target based on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic positioning result of the in-vehicle target includes: performing target positioning on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic positioning coordinates of the in-vehicle target; determining the confidence score corresponding to the acoustic positioning coordinates according to the sound quality index of the in-vehicle sound data.

[0013] In some implementations, the sound quality index includes the signal-to-noise ratio and the multipath interference index, and the determining the confidence score corresponding to the acoustic positioning coordinates according to the sound quality index of the in-vehicle sound data includes: weighting the signal-to-noise ratio and the multipath interference index according to the weight parameter corresponding to the signal-to-noise ratio and the weight parameter corresponding to the multipath interference index to obtain the confidence score corresponding to the acoustic positioning coordinates.

[0014] In some implementations, the visual positioning result includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates, the acoustic positioning result includes the acoustic positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, and the determining the in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result includes: fusing the visual positioning coordinates and the acoustic positioning coordinates according to the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates to obtain the fused coordinates corresponding to the in-vehicle target; determining the in-vehicle position information according to the fused coordinates.

[0015] In some implementations, the fusing of the visual positioning coordinates and the acoustic positioning coordinates based on the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates to obtain the fused coordinates corresponding to the in-vehicle target includes: when the confidence score corresponding to the visual positioning coordinates is greater than a first score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than a second score threshold, weighting the visual positioning coordinates and the acoustic positioning coordinates according to the weight parameter corresponding to the visual positioning coordinates and the weight parameter corresponding to the acoustic positioning coordinates to obtain the fused coordinates; when the confidence score corresponding to the visual positioning coordinates is less than a third score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than a fourth score threshold, determining the fused coordinates as the acoustic positioning coordinates.

[0016] In some implementations, the determining of the in-vehicle position information based on the fused coordinates includes: when there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates, correcting the fused coordinates according to the historical trajectory information of the in-vehicle target in the past time to obtain the corrected coordinates, where the in-vehicle position information includes the corrected coordinates.

[0017] In some implementations, before correcting the fused coordinates according to the trajectory information of the in-vehicle target in the past time to obtain the corrected coordinates when there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates, it further includes: determining whether there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates according to the Euclidean distance between the visual positioning coordinates and the acoustic positioning coordinates.

[0018] In some implementations, after determining the in-vehicle position information of the in-vehicle target based on the visual positioning result and the acoustic positioning result, it further includes: updating the trajectory information of the in-vehicle target according to the in-vehicle position information.

[0019] According to a second aspect of the embodiments of the present application, there is provided an in-vehicle target positioning device, including: a data acquisition unit that acquires in-vehicle visual data and in-vehicle sound data; a spatio-temporal synchronization unit that is used to perform spatio-temporal synchronization on the in-vehicle visual data and the in-vehicle sound data; a visual positioning unit that is used to perform positioning of the in-vehicle target based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual positioning result of the in-vehicle target; an acoustic positioning unit that is used to perform positioning of the in-vehicle target based on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic positioning result of the in-vehicle target; and a position determination unit that is used to determine the in-vehicle position information of the in-vehicle target based on the visual positioning result and the acoustic positioning result.

[0020] According to a third aspect of the embodiments of the present application, an electronic device is provided, including a memory and a processor; the memory is connected to the processor and is used for storing a program; the processor is used for implementing the in-vehicle target positioning method as described in the first aspect or any implementation manner of the first aspect by running the program in the memory.

[0021] According to a fourth aspect of the embodiments of the present application, a computer program product is provided, and when the computer program is executed by a processor, it implements the in-vehicle target positioning method as described in the first aspect or any implementation manner of the first aspect.

[0022] According to a fifth aspect of the embodiments of the present application, a storage medium is provided, on which a computer program is stored, and when the computer program is run by a processor, it implements the in-vehicle target positioning method as described in the first aspect or any implementation manner of the first aspect.

[0023] An in-vehicle target positioning method, device, equipment and medium provided by the embodiments of the present application perform spatio-temporal synchronization on in-vehicle visual data and in-vehicle sound data, and respectively perform positioning of in-vehicle targets according to the spatio-temporally synchronized in-vehicle visual data and the spatio-temporally synchronized in-vehicle sound data to obtain a visual positioning result and an acoustic positioning result of the in-vehicle target, and determine the in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result. On the one hand, based on the in-vehicle visual data and the in-vehicle sound data, in-vehicle target positioning based on multi-modal data is realized. On the other hand, through the time synchronization of the in-vehicle visual data and the in-vehicle sound data, the combination effect of the positioning results of the multi-modal data is improved, thereby improving the accuracy of in-vehicle target positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required to be used in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained according to the provided drawings without creative efforts.

[0025] Figure 1 is the flow of an in-vehicle target positioning method provided by an embodiment of the present application Figure 1 ;

[0026] Figure 2 is an example diagram for performing PTP synchronization on in-vehicle visual data and in-vehicle sound data;

[0027] Figure 3 is the flow of an in-vehicle target positioning method provided by an embodiment of the present application Figure 2 ;

[0028] Figure 4 It is a schematic structural diagram of an in-vehicle target positioning device provided by an embodiment of the present application;

[0029] Figure 5 It is a schematic structural diagram of a vehicle-mounted device provided by an embodiment of the present application;

[0030] Figure 6 It is a schematic structural diagram of a vehicle provided by an embodiment of the present application. Specific embodiments

[0031] The technical solution proposed by the embodiment of the present application is applicable to the vehicle driving scenario, aiming to significantly improve the accuracy of in-vehicle target positioning by improving the in-vehicle target positioning method. By adopting the technical solution described in the embodiment of the present application, it is possible to realize in-vehicle target positioning based on multi-modal data and improve the combination effect of the positioning results corresponding to the multi-modal data respectively, thereby improving the accuracy of in-vehicle target positioning.

[0032] Next, the technical solution in the embodiment of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiment of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0033] Before introducing the solution of the present application, the related technologies will be introduced first:

[0034] In one approach, target positioning is performed based on visual data. However, the in-vehicle environment has the characteristics of relatively dim light and easy occlusion of targets, so the accuracy of in-vehicle target positioning based on visual data is relatively low. In another approach, target positioning is performed based on sound data. However, the in-vehicle sound field is relatively complex and there is a lot of noise (such as music played in the vehicle and tire noise of the vehicle), so target positioning based on sound data is easily interfered by noise and the accuracy of in-vehicle target positioning is relatively low.

[0035] It can be seen that for the in-vehicle scenario, the reliability of single-modal data is relatively low, and the accuracy of target positioning based on single-modal data is relatively low. It is possible to consider in-vehicle target positioning based on multi-modal data. In this process, if the multi-modal data is simply superimposed without considering the spatio-temporal misalignment before the multi-modal data, the improvement effect on the accuracy of in-vehicle target positioning is limited, and it may also cause delays in in-vehicle target positioning, affecting the timeliness of in-vehicle target positioning.

[0036] In view of this, the embodiments of the present application are committed to providing a method, device, equipment and medium for in-vehicle target positioning. The multi-modal data used includes in-vehicle visual data and in-vehicle sound data. Before performing in-vehicle target positioning based on the in-vehicle visual data and in-vehicle sound data, the in-vehicle visual data and in-vehicle sound data are first synchronized in space and time to solve the problem of spatio-temporal misalignment between the in-vehicle visual data and the in-vehicle sound data, and to avoid the adverse effects of this spatio-temporal misalignment problem on the accuracy and timeliness of in-vehicle target positioning. Then, based on the spatially and temporally synchronized in-vehicle visual data and the spatially and temporally synchronized in-vehicle sound data, in-vehicle target positioning is performed respectively to obtain the visual positioning result and the acoustic positioning result of the in-vehicle target. According to the visual positioning result and the acoustic positioning result, the in-vehicle position information of the in-vehicle target is determined. Thus, in-vehicle target positioning based on multi-modal data is realized, effectively improving the accuracy and timeliness of in-vehicle target positioning.

[0037] Among them, the method, device, equipment and medium for in-vehicle target positioning provided by the embodiments of the present application can be applied to a variety of scenarios: in-vehicle device interaction scenarios (such as voice interaction scenarios of in-vehicle devices such as in-vehicle air conditioners, in-vehicle seats, in-vehicle screens, in-vehicle lights, etc. In the voice interaction scenarios of in-vehicle devices, by positioning the in-vehicle target that issues a voice command, the in-vehicle devices near the in-vehicle target are adjusted to improve the accuracy of in-vehicle device adjustment), in-vehicle safety protection scenarios (such as, during vehicle driving, positioning the in-vehicle target to identify whether the in-vehicle target is on the seat to remind the in-vehicle target to sit properly, or, after the vehicle stops, positioning the in-vehicle target to identify whether the in-vehicle target is left in the vehicle), in-vehicle personnel rescue scenarios (such as, by positioning the in-vehicle personnel to determine the specific location of the in-vehicle personnel to implement rescue faster and more accurately), in-vehicle personnel health monitoring scenarios (such as, a health monitoring system is configured on the vehicle, and by positioning the in-vehicle personnel, the vital signs of the in-vehicle personnel are accurately monitored). The above scenarios are only part of the scenarios, and the embodiments of the present application can also be applied to other scenarios, which will not be exemplified one by one here.

[0038] It should be noted that the driving planning method provided by the embodiments of the present application can be executed on an in-vehicle device or on a device that establishes a communication connection with the vehicle. The device that establishes a communication connection with the vehicle is, for example, a terminal used by an in-vehicle user (such as a mobile phone), a device that provides remote services for the vehicle.

[0039] Exemplary method

[0040] Figure 1 The flowchart of a method for in-vehicle target positioning provided by the embodiments of the present application Figure 1 As Figure 1 shown, the method for in-vehicle target positioning provided in this embodiment includes the following steps S101 to step S104:

[0041] S101, obtain in-vehicle visual data and in-vehicle sound data.

[0042] Among them, the in-vehicle visual data and the in-vehicle sound data are obtained by collecting visual data and sound data inside the same vehicle.

[0043] In one example, the in-vehicle visual data and the in-vehicle sound data are the original visual data and the original sound data collected inside the same vehicle.

[0044] In another example, the in-vehicle visual data and the in-vehicle sound data are obtained by preprocessing the original visual data and the original sound data collected inside the same vehicle. The preprocessing operations include, for example, data cleaning, data denoising, data enhancement, etc.

[0045] Among them, the in-vehicle visual data is image data.

[0046] In one example, the in-vehicle visual data includes one or more segments of in-vehicle videos.

[0047] In another example, the in-vehicle visual data includes in-vehicle images or in-vehicle videos captured when the in-vehicle target positioning condition is triggered. The in-vehicle target positioning condition can be determined according to the application scenario of in-vehicle target positioning. For example, the in-vehicle target positioning condition is that the vehicle driving speed is greater than the speed threshold, the vehicle jitter degree is greater than the degree threshold, the vehicle has a collision, etc.

[0048] Among them, the in-vehicle sound data may include one or more segments of in-vehicle audio.

[0049] In this embodiment, the in-vehicle visual data sent by the in-vehicle visual sensor can be obtained, and the in-vehicle sound data sent by the in-vehicle acoustic sensor can be obtained. The in-vehicle visual data is the original visual data collected by the visual sensor, and the in-vehicle sound data is the original sound data collected by the acoustic sensor; or, the in-vehicle visual data and the in-vehicle sound data can be obtained from a storage device (such as the memory on the vehicle-mounted device). The storage device can store the original visual data and the original acoustic data, or can store the preprocessing data of the original visual data and the preprocessing data of the original acoustic data.

[0050] In one example, the visual sensor can adopt a wide-angle infrared camera. On the one hand, it can improve the acquisition range of the in-vehicle visual data and achieve full coverage of the in-vehicle seats. On the other hand, it can improve the acquisition quality of the in-vehicle visual data in low-light conditions (such as night environment).

[0051] In one example, the acoustic sensor may adopt a distributed microphone array. Multiple microphones in the distributed microphone array collect sound signals simultaneously, which have high sensitivity to the perception of in-vehicle sounds and can collect sounds emitted by in-vehicle targets at different positions, thereby improving the sound localization accuracy.

[0052] S102, perform spatio-temporal synchronization on the in-vehicle visual data and the in-vehicle sound data.

[0053] In this embodiment, the in-vehicle visual data and the in-vehicle sound data are collected by different sensors. Therefore, there is a spatio-temporal misalignment between the in-vehicle visual data and the in-vehicle sound data. Spatio-temporal misalignment refers to a situation where there is inconsistency or mismatch in time and space. It can be understood that the in-vehicle visual sensor and the in-vehicle acoustic sensor have their own independent clock systems, and it is difficult for their clock systems to achieve complete synchronization, resulting in the inability to align the video frames in the in-vehicle visual data with the audio frames in the in-vehicle sound data in time; there are differences in the installation positions and data collection perspectives between the in-vehicle visual sensor and the in-vehicle acoustic sensor, resulting in the inability to align the in-vehicle visual data with the in-vehicle sound data in space. Based on the clock difference between the in-vehicle visual sensor and the in-vehicle acoustic sensor, time synchronization can be performed on the in-vehicle visual data and the in-vehicle sound data. Or, the correlation between the in-vehicle visual data and the in-vehicle sound data in the time dimension can be captured through an algorithm to achieve time synchronization of the in-vehicle visual data and the in-vehicle sound data; based on the spatial conversion relationship between the in-vehicle visual sensor and the in-vehicle acoustic sensor, time synchronization can be performed on the in-vehicle visual data and the in-vehicle sound data. Or, the correlation between the in-vehicle visual data and the in-vehicle sound data in the spatial dimension can be captured through an algorithm to achieve spatial synchronization of the in-vehicle visual data and the in-vehicle sound data, obtaining the spatio-temporally synchronized in-vehicle visual data and the spatio-temporally synchronized in-vehicle sound data. Among them, time synchronization of the in-vehicle visual data and the in-vehicle sound data can be performed first, and then spatial synchronization of the in-vehicle visual data and the in-vehicle sound data can be performed. Or, spatial synchronization of the in-vehicle visual data and the in-vehicle sound data can be performed first, and then time synchronization of the in-vehicle visual data and the in-vehicle sound data can be performed.

[0054] Among them, the spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data can be carried out during the acquisition process of the in-vehicle visual data and the in-vehicle sound data. For example, during the process of collecting the in-vehicle visual data through the in-vehicle visual sensor and collecting the in-vehicle sound data through the in-vehicle acoustic sensor, spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data is performed. The spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data can also be carried out after the acquisition of the in-vehicle visual data and the in-vehicle sound data.

[0055] S103, perform localization of the in-vehicle target based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual localization result of the in-vehicle target.

[0056] Among them, the in-vehicle targets may include in-vehicle personnel, in-vehicle animals, and / or in-vehicle items.

[0057] Among them, the visual positioning result of the in-vehicle target may include the visual positioning coordinates of the in-vehicle target, and the visual positioning coordinates of the in-vehicle target indicate the position of the in-vehicle target that appears in the in-vehicle visual data.

[0058] In this embodiment, a target detection algorithm for visual data may be used to perform target detection on the in-vehicle visual data after spatio-temporal synchronization to obtain the visual positioning result of the in-vehicle target.

[0059] S104. Perform positioning of the in-vehicle target based on the in-vehicle sound data after spatio-temporal synchronization to obtain the acoustic positioning result of the in-vehicle target.

[0060] Among them, the acoustic positioning result of the in-vehicle target may include the acoustic positioning coordinates of the in-vehicle target, and the acoustic positioning coordinates of the in-vehicle target indicate the sound source position of the in-vehicle sound data.

[0061] In this embodiment, a target positioning algorithm for sound data may be used to perform sound source positioning on the in-vehicle sound data after spatio-temporal synchronization to obtain the sound positioning result of the in-vehicle target.

[0062] S105. Determine the in-vehicle position information of the in-vehicle target according to the visual positioning result of the in-vehicle target and the acoustic positioning result of the in-vehicle target.

[0063] In this embodiment, a set fusion method may be used to fuse the visual positioning result of the in-vehicle target and the acoustic positioning result of the in-vehicle target to obtain the in-vehicle position information of the in-vehicle target. Compared with single in-vehicle visual positioning or single in-vehicle sound positioning, this solution combines in-vehicle visual positioning and in-vehicle sound positioning to improve the accuracy of in-vehicle target positioning.

[0064] In the embodiment of the present application, spatio-temporal synchronization is performed on the in-vehicle visual data and the in-vehicle sound data, and in-vehicle target positioning is performed based on the in-vehicle visual data after spatio-temporal synchronization and the in-vehicle sound data after spatio-temporal synchronization to obtain the visual positioning result of the in-vehicle target and the acoustic positioning result of the in-vehicle target, and the in-vehicle position information of the in-vehicle target is determined by combining the visual positioning result and the acoustic positioning result. Thus, on the premise of solving the spatio-temporal misalignment problem of multi-modal data, in-vehicle target positioning based on multi-modal data is realized, and the accuracy and timeliness of in-vehicle target positioning are improved.

[0065] In some embodiments, in the spatio-temporal synchronization of in-vehicle visual data and in-vehicle sound data, the process of time synchronization between in-vehicle visual data and in-vehicle sound data includes: establishing a unified timestamp for in-vehicle visual data and in-vehicle sound data to obtain time-synchronized in-vehicle visual data and time-synchronized in-vehicle sound data. After time synchronization, the timestamps on the time axis corresponding to the in-vehicle visual data are consistent with the timestamps on the time axis corresponding to the in-vehicle sound data, achieving the alignment of the in-vehicle visual data and the in-vehicle sound data in terms of timestamps and improving the accuracy of time synchronization.

[0066] Among them, there are differences in time accuracy between the clock systems of the visual sensors in the vehicle and the clock systems of the acoustic sensors in the vehicle, making it difficult to achieve complete synchronization, resulting in misalignment between the timestamps corresponding to the video frames in the in-vehicle visual data and the timestamps corresponding to the audio frames in the in-vehicle sound data.

[0067] Suppose that when the visual sensors in the vehicle mark video frames and the acoustic sensors in the vehicle mark audio frames, both use coordinated universal time (UTC). Due to the difference in the accuracy of the clock systems between the video sensor and the visual sensor, for the same physical event, the timestamp recorded by the visual sensor in the vehicle is T = 1000 milliseconds (ms), and the timestamp recorded by the acoustic sensor in the vehicle is T = 1003 ms. This deviation between the two timestamps cannot be ignored in scenarios that require high-precision time synchronization (such as positioning scenarios and emergency braking scenarios), and forced alignment of the timestamps is required.

[0068] In this embodiment, a unified timestamp can be established for in-vehicle visual data and in-vehicle sound data through a hardware-level time synchronization processing method and / or a software-level time synchronization processing method to obtain time-synchronized in-vehicle visual data and time-synchronized in-vehicle sound data.

[0069] In one example, establishing a unified timestamp for in-vehicle visual data and in-vehicle sound data through a hardware-level time synchronization processing method and a software-level time synchronization processing method may include: before acquiring in-vehicle visual data and in-vehicle sound data, connecting the visual sensors in the vehicle and the acoustic sensors in the vehicle to a network that supports the precision time protocol (PTP); determining the grandmaster clock and the slave clock in the visual sensors and the acoustic sensors in the vehicle; calibrating the slave clock based on the grandmaster clock to obtain the PTP-synchronized slave clock; and during the process of acquiring in-vehicle visual data and in-vehicle sound data, determining consistent timestamps for the video frames in the in-vehicle visual data and the audio frames in the in-vehicle sound data according to the grandmaster clock and the PTP-synchronized slave clock.

[0070] Among them, PTP is a protocol for achieving high-precision time synchronization in a network. PTP uses a master-slave clock architecture for time synchronization, taking the master clock as the time reference to calibrate the local time of the slave clock to maintain the synchronization between the master clock and the slave clock.

[0071] In this example, first, the in-vehicle vision sensor and the in-vehicle acoustic sensor are connected to the same network that supports PTP, enabling the in-vehicle vision sensor and the in-vehicle acoustic sensor to communicate with each other and perform time synchronization according to PTP. Then, the time accuracy of the in-vehicle vision sensor and the in-vehicle acoustic sensor can be compared to obtain a comparison result; based on the comparison result, the master clock and the slave clock are determined among the in-vehicle vision sensor and the in-vehicle acoustic sensor. Specifically, if the time accuracy of the in-vehicle vision sensor is higher than that of the in-vehicle acoustic sensor, the master clock can be determined as the in-vehicle vision sensor and the slave clock as the in-vehicle acoustic sensor; otherwise, the master clock can be determined as the in-vehicle acoustic sensor and the slave clock as the in-vehicle vision sensor. After that, the slave clock can be periodically calibrated based on the master clock. Specifically, taking the master clock as the time reference, it can periodically send time synchronization messages to the slave clock through the network that supports PTP, and the slave clock calibrates its local time according to the clock synchronization message to keep in sync with the master clock, obtaining the slave clock after PTP synchronization; timestamps can be added to the video frames in the in-vehicle vision data and the audio frames in the in-vehicle sound data according to the master clock and the slave clock after PTP synchronization.

[0072] For example, taking the in-vehicle video data as a video frame sequence and the in-vehicle sound data as sound data packets as an example, Figure 2 It is an example diagram for PTP synchronization of the video frame sequence and the sound data packets, as Figure 2 shown, the timestamp of the first video frame is 3 ms, and the timestamp of the second video frame is 24 ms; after PTP synchronization, the same timestamp as the first video frame is established for Packet 1 in the sound data packets, that is, the timestamp of Packet 1 and the timestamp of the first video frame are both 3 ms, and the same timestamp as the second video frame is established for Packet 2 in the sound data packets, that is, the timestamp of Packet 2 and the timestamp of the second video frame are both 24 ms.

[0073] In another example, through a hardware-level time synchronization processing method, a unified timestamp is established for the in-vehicle vision data and the in-vehicle sound data, including: during the process of acquiring the in-vehicle vision data and the in-vehicle sound data, according to the clock synchronization module deployed on the vehicle (such as a high-precision clock chip), timestamps are added to the video frames in the in-vehicle vision data and the audio frames in the in-vehicle sound data.

[0074] In another example, through a software-level time synchronization processing method, a unified timestamp is established for in-vehicle visual data and in-vehicle audio data, including: obtaining the time correspondence between in-vehicle visual data and in-vehicle audio data by performing feature matching on the in-vehicle visual data and in-vehicle audio data, and establishing a unified timestamp for the in-vehicle visual data and in-vehicle audio data according to the time correspondence between the in-vehicle visual data and in-vehicle audio data. For example: the lip movement feature of an in-vehicle target is extracted from the in-vehicle visual data, and the voice feature of the in-vehicle target is extracted from the in-vehicle audio data; by matching the lip movement feature with the voice feature, the correspondence between the lip movement feature and the voice feature is established, and the same timestamp is established for the lip movement feature and the voice feature corresponding to the lip movement feature.

[0075] In some embodiments, in the spatio-temporal synchronization of in-vehicle visual data and in-vehicle audio data, the spatial synchronization process of in-vehicle visual data and in-vehicle audio data includes: converting the time-synchronized in-vehicle visual data from the visual sensor coordinate system to a set in-vehicle coordinate system to obtain the spatio-temporally synchronized in-vehicle visual data; converting the time-synchronized in-vehicle audio data from the acoustic sensor coordinate system to the in-vehicle coordinate system to obtain the spatio-temporally synchronized in-vehicle audio data. Thus, by mapping the in-vehicle visual data and in-vehicle audio data to the same spatial framework (i.e., the same in-vehicle coordinate system), the spatial synchronization of in-vehicle visual data and in-vehicle audio data is achieved.

[0076] Among them, the set in-vehicle coordinate system is a unified spatial reference for the spatial synchronization of in-vehicle visual data and in-vehicle audio data, and the in-vehicle coordinate system can be a three-dimensional rectangular coordinate system, such as a Cartesian coordinate system.

[0077] Optionally, the origin of the in-vehicle coordinate system is the center point of the vehicle steering wheel, the center point of the driver's seat, or the center point of the vehicle interior space.

[0078] In this embodiment, in an actual scenario, the visual sensor and the acoustic sensor in the vehicle have their own coordinate systems, resulting in a spatial misalignment problem between the in-vehicle visual data and the in-vehicle sound data. To solve this spatial misalignment problem, first, a unified in-vehicle coordinate system can be defined as the reference coordinate system for the spatial synchronization of the in-vehicle visual data and the in-vehicle sound data; then, through a calibration method, the conversion relationship (including the rotation matrix and the translation matrix) between the visual sensor coordinate system and the in-vehicle coordinate system and the conversion relationship (including the rotation matrix and the translation matrix) between the acoustic sensor coordinate system and the in-vehicle coordinate system are determined; after that, during the process of spatial synchronization, for the video frames in the in-vehicle visual data after time synchronization, according to the conversion relationship between the visual sensor coordinate system and the in-vehicle coordinate system, the pixel points on the video frames are converted from the visual sensor coordinate system to the in-vehicle coordinate system to obtain the video frames after coordinate transformation, and the video frames after coordinate transformation form the in-vehicle visual data after spatio-temporal synchronization; for the audio frames in the in-vehicle sound data after time synchronization, according to the conversion relationship between the acoustic sensor coordinate system and the in-vehicle coordinate system, the signal points on the audio frames are converted from the acoustic sensor to the in-vehicle coordinate system to obtain the audio frames after coordinate transformation, and the audio frames after coordinate transformation form the in-vehicle sound data after spatio-temporal synchronization.

[0079] It should be noted that in the above example, time synchronization is performed first and then spatial synchronization, but it is also possible to perform spatial synchronization first and then time synchronization, which will not be elaborated here.

[0080] Figure 3 The flowchart of an in-vehicle target positioning method provided by an embodiment of the present application Figure 2 As Figure 3 shown, the in-vehicle target positioning method provided by this embodiment includes the following steps S301 to S305:

[0081] S301, obtain in-vehicle visual data and in-vehicle sound data.

[0082] S302, perform spatio-temporal synchronization on the in-vehicle visual data and the in-vehicle sound data.

[0083] Among them, the implementation principles and technical effects of S301 to S302 refer to the foregoing embodiments and will not be elaborated here.

[0084] S303, perform positioning of in-vehicle targets based on the in-vehicle visual data after spatio-temporal synchronization to obtain the visual positioning results of the in-vehicle targets, and the visual positioning results include the visual positioning coordinates of the in-vehicle targets and the confidence scores corresponding to the visual positioning coordinates.

[0085] Among them, the confidence score corresponding to the visual positioning coordinates is an index used to measure the reliability and accuracy of the visual positioning coordinates. For example, the higher the confidence score corresponding to the visual positioning coordinates, the higher the reliability and accuracy of the visual positioning coordinates.

[0086] In this embodiment, the object detection algorithm can be used to perform object detection on the in-vehicle visual data after spatio-temporal synchronization, so as to obtain the visual positioning result of the in-vehicle object, which includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates.

[0087] In a possible implementation manner, S303 includes: S3031, performing object detection on the in-vehicle visual data after spatio-temporal synchronization to obtain the visual positioning coordinates of the in-vehicle object and the visual detection score corresponding to the visual positioning coordinates; S3032, determining the confidence score corresponding to the visual positioning coordinates according to the optical radiation index and the visual detection score of the in-vehicle visual data.

[0088] In S3031, the video frames in the in-vehicle visual data after spatio-temporal synchronization can be input into the object detection algorithm, and the object detection algorithm is used to perform in-vehicle object detection on the video frames to obtain the visual positioning coordinates of the in-vehicle object and the visual detection score corresponding to the visual positioning coordinates.

[0089] Optionally, the object detection algorithm is the YOLO (you only look once) algorithm. Through the YOLO algorithm, the bounding box of the in-vehicle object can be detected in the video frame, and the coordinates of the bounding box and the confidence score corresponding to the bounding box can be obtained. The confidence score corresponding to the bounding box represents the probability that there is an in-vehicle object within the bounding box. Among them, the coordinates of the bounding box of the in-vehicle object are the visual positioning coordinates of the in-vehicle object, and the confidence score corresponding to the bounding box is the visual detection score corresponding to the visual positioning coordinates. Thus, the accuracy of visual positioning is improved through the YOLO algorithm.

[0090] In S3032, considering that the reliability of video positioning is different under different lighting conditions, the visual detection score corresponding to the visual positioning coordinates is adjusted based on the optical radiation index of the in-vehicle visual data to obtain the final confidence score corresponding to the visual positioning coordinates. Thus, based on the in-vehicle environment situation, the confidence score of the visual positioning coordinates is dynamically adjusted to improve the accuracy of the confidence score.

[0091] Among them, the optical radiation index can be collected by sensors deployed on the vehicle.

[0092] Optionally, the optical radiation index includes illuminance and / or luminance. Among them, illuminance and luminance are important indexes affecting the acquisition quality of in-vehicle visual data.

[0093] In one example, according to the weight parameter corresponding to the optical radiation index and the weight parameter corresponding to the visual detection score, the optical radiation index and the visual detection score corresponding to the visual positioning coordinates are weighted to obtain the confidence score corresponding to the visual positioning coordinates. Among them, the weight parameter corresponding to the optical radiation index and the weight parameter corresponding to the visual detection score can be determined through experiments. By reasonable weight parameters, the rationality of the influence of the optical radiation index on visual positioning is improved, and the accuracy of the confidence score is improved.

[0094] In another example, if the optical radiation index is greater than the first index threshold, the visual detection score corresponding to the visual positioning coordinates is increased by a first value to obtain the confidence score corresponding to the visual positioning coordinates; if the optical radiation index is less than the second index threshold, the visual detection score corresponding to the visual positioning coordinates is decreased by a second value to obtain the confidence score corresponding to the visual positioning coordinates. Among them, the first index threshold is greater than the second index threshold, and the magnitude relationship between the first value and the second value is not limited.

[0095] S304. According to the in-vehicle sound data after spatio-temporal synchronization, perform the positioning of the in-vehicle target to obtain the acoustic positioning result of the in-vehicle target. The acoustic positioning result includes the acoustic positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates.

[0096] Among them, the confidence score corresponding to the acoustic positioning coordinates is an index used to measure the reliability and accuracy of the acoustic positioning coordinates. For example, the higher the confidence score corresponding to the acoustic positioning coordinates, the higher the reliability and accuracy of the acoustic positioning coordinates.

[0097] In this embodiment, an acoustic positioning algorithm can be used to perform sound source positioning on the in-vehicle sound data after spatio-temporal synchronization to obtain the acoustic positioning result of the in-vehicle target, which includes the acoustic positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates.

[0098] In a possible implementation manner, S304 includes: S3041. Perform target positioning on the in-vehicle sound data after spatio-temporal synchronization to obtain the acoustic positioning coordinates of the in-vehicle target; S3042. Determine the confidence score corresponding to the acoustic positioning coordinates according to the sound quality index of the in-vehicle sound data.

[0099] In S3041, an acoustic positioning algorithm can be used to perform sound source positioning on the in-vehicle sound data after spatio-temporal synchronization to obtain the acoustic positioning coordinates of the in-vehicle target.

[0100] Optionally, the acoustic positioning algorithm is the time difference of arrival (TDOA) algorithm. The in-vehicle sound sensor may include multiple sub-sensors. For example, the in-vehicle sound sensor is a distributed microphone array, which can determine the time difference of the same sound signal arriving at different sub-sensors based on the in-vehicle sound data, and perform sound source positioning on the sound signal according to the time difference. Thus, the accuracy of acoustic positioning is improved through the TDOA algorithm.

[0101] In S3042, considering that the reliability of acoustic positioning is different under different sound qualities, the confidence score corresponding to the acoustic positioning coordinates is determined based on the sound quality index of the in-vehicle sound data to improve the accuracy of the confidence score corresponding to the acoustic positioning coordinates. Specifically, the confidence score corresponding to the acoustic positioning coordinates can be calculated by using a formula-based calculation method based on the sound quality index of the in-vehicle sound data, or, for the in-vehicle acoustic positioning score, the corresponding relationship between the sound quality index and the confidence score is preset in advance, and the confidence score found in this corresponding relationship with the sound quality index of the in-vehicle sound data as the key information is the confidence score corresponding to the acoustic positioning coordinates.

[0102] Among them, the acoustic quality index can be obtained by performing quality analysis on the in-vehicle sound data.

[0103] Optionally, the acoustic quality index includes the signal-to-noise ratio and / or the multipath interference index. The signal-to-noise ratio of the in-vehicle sound data reflects the relative intensity of the useful signal and the noise in the in-vehicle sound data. In the in-vehicle environment, the sound signal propagates through multiple paths, and the multipath interference index of the in-vehicle sound data can reflect the interference situation between the multipath propagations of the in-vehicle sound data, and thus reflect the sound quality.

[0104] In this optional method, the power of the effective signal segment can be extracted from the in-vehicle sound data; the noise power can be estimated based on the signal silent segment (the signal segment without in-vehicle target sound) included in the in-vehicle sound data, or the noise power of the in-vehicle sound data can be estimated through a noise estimation model to obtain the power of the noise signal segment; the signal-to-noise ratio of the in-vehicle sound data is calculated based on the power of the effective signal segment and the power of the noise signal segment. The multipath interference index of the in-vehicle sound data can be obtained through the multipath interference evaluation of the in-vehicle sound data.

[0105] Furthermore, the multipath interference index includes the multipath energy ratio. In particular, it may include the multipath energy ratio of the direct path, that is, the ratio of the energy of the direct path to the total received energy.

[0106] In addition to the multipath energy ratio, the multipath interference index may include one or more of the following indexes: delay spread (reflecting the dispersion of the sound signal in time caused by multipath interference), power delay profile (reflecting the relative intensity of different delay components in the multipath signal), fading characteristic indexes (such as fading depth, fading rate, average fading time), etc., which will not be described one by one here.

[0107] In a possible implementation, the sound quality index includes the signal-to-noise ratio and the multipath interference index. S3042 includes: weighting the signal-to-noise ratio and the multipath interference index according to the weight parameter corresponding to the signal-to-noise ratio and the weight parameter corresponding to the multipath interference index to obtain the confidence score corresponding to the acoustic positioning coordinate. Among them, the weight parameter corresponding to the signal-to-noise ratio and the weight parameter corresponding to the multipath interference index can be determined through experiments. By reasonable weight parameters, the rationality of the influence of the signal-to-noise ratio and the multipath interference index on acoustic positioning is improved, and the accuracy of the confidence score corresponding to the acoustic positioning coordinate is improved.

[0108] For example, the calculation formula of the confidence score corresponding to the acoustic positioning coordinate is expressed as:

[0109] Ca = α·C SNR +β·R mp

[0110] Where C SNR represents the signal-to-noise ratio, R mp represents the multipath energy ratio, α represents the weight parameter corresponding to the signal-to-noise ratio, β represents the weight parameter corresponding to the multipath interference index, and Ca represents the confidence score corresponding to the acoustic positioning coordinate. For example, α is equal to 0.7 and β is equal to 0.3.

[0111] Furthermore, according to the signal-to-noise ratio or the multipath interference index, it can be determined whether the current in-vehicle scenario is a low signal-to-noise ratio scenario or a high multipath interference scenario. If the current in-vehicle scenario is a low signal-to-noise ratio scenario or a high multipath interference scenario, the confidence score corresponding to the acoustic positioning coordinate is exponentially decayed, so that the confidence score under extreme conditions drops rapidly, and the accuracy of the confidence score is improved.

[0112] For example, when the signal-to-noise ratio is less than the third index threshold, it is determined that the current in-vehicle scenario is a low signal-to-noise ratio scenario. When the multipath interference index is greater than the fourth index threshold, it is determined that the current in-vehicle scenario is a multipath interference scenario. The magnitudes of the third index threshold and the fourth index threshold are not limited.

[0113] S305, determine the in-vehicle position information of the in-vehicle target according to the visual positioning result of the in-vehicle target and the acoustic positioning result of the in-vehicle target.

[0114] In this embodiment, based on the visual positioning coordinates, the confidence score corresponding to the visual positioning coordinates, the acoustic positioning coordinates, and the confidence score corresponding to the acoustic positioning coordinates, the in-vehicle position information of the in-vehicle target is determined, and the in-vehicle position information includes the final in-vehicle coordinates of the in-vehicle target.

[0115] In a possible implementation, S305 includes: S3051, based on the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, fusing the visual positioning coordinates and the acoustic positioning coordinates to obtain the fused coordinates corresponding to the in-vehicle target; S3052, based on the fused coordinates, determining the in-vehicle position information of the in-vehicle target.

[0116] In an implementation of S3051, the weight parameter corresponding to the visual positioning coordinates can be determined according to the confidence score corresponding to the visual positioning coordinates. This weight parameter can be the confidence score corresponding to the visual positioning coordinates or a value determined (such as calculated) based on this confidence score; the weight parameter corresponding to the acoustic positioning coordinates is determined according to the confidence score corresponding to the acoustic positioning coordinates. This weight parameter can be the confidence score corresponding to the acoustic positioning coordinates or a value determined (such as calculated) based on this confidence score; based on the weight parameter corresponding to the visual positioning coordinates and the weight parameter corresponding to the acoustic positioning coordinates, the visual positioning coordinates and the acoustic positioning coordinates are weighted to obtain the fused coordinates corresponding to the in-vehicle target.

[0117] In another implementation of S3051, according to the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, among a set of multiple fusion rules, the target fusion rule that the visual positioning coordinates and the acoustic positioning coordinates conform to is determined; according to the target fusion rule, the visual positioning coordinates and the acoustic positioning coordinates are fused to obtain the fused coordinates corresponding to the in-vehicle target.

[0118] In an example, S3051 includes: when the confidence score corresponding to the visual positioning coordinates is greater than the first score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the second score threshold, based on the weight parameter corresponding to the visual positioning coordinates and the weight parameter corresponding to the acoustic positioning coordinates, the visual positioning coordinates and the acoustic positioning coordinates are weighted to obtain the fused coordinates; when the confidence score corresponding to the visual positioning coordinates is less than the third score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the fourth score threshold, the fused coordinates are determined to be the acoustic positioning coordinates.

[0119] Among them, the first score threshold, the second score threshold, the third score threshold, and the fourth score threshold are pre-set thresholds and can be determined through multiple experiments.

[0120] In this example, the confidence score corresponding to the visual positioning coordinates is greater than the first score threshold, indicating that the optical conditions inside the vehicle are good, and the error between the visual positioning coordinates and the actual coordinates of the target inside the vehicle is small. The confidence score corresponding to the acoustic positioning coordinates is greater than the second score threshold, indicating that the influence of environmental interference on the acoustic positioning algorithm is controllable. Therefore, when the confidence score corresponding to the visual positioning coordinates is greater than the first score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the second score threshold, the visual positioning coordinates and the acoustic positioning coordinates are weighted according to the weight parameters corresponding to the visual positioning coordinates and the weight parameters corresponding to the acoustic positioning coordinates to obtain the fused coordinates.

[0121] Optionally, the weight parameter corresponding to the visual positioning coordinates is greater than the weight parameter corresponding to the acoustic positioning coordinates. When the optical conditions inside the vehicle are good, the accuracy of the visual positioning coordinates is greater than that of the acoustic positioning coordinates. Compared with the acoustic positioning coordinates, assigning a larger weight parameter to the visual positioning coordinates is beneficial to improving the accuracy of the target positioning inside the vehicle.

[0122] In this example, the confidence score corresponding to the visual positioning coordinates is less than the third score threshold, indicating that the optical conditions inside the vehicle are poor, and the reliability and accuracy of the visual positioning coordinates are greatly reduced; the confidence score corresponding to the acoustic positioning coordinates is greater than the fourth score threshold, indicating that the reliability and accuracy of the acoustic positioning coordinates are relatively high. Therefore, the fused coordinates are determined to be the acoustic positioning coordinates.

[0123] Optionally, the third score threshold can be less than the first score threshold, and the fourth score can be greater than the second score threshold, so as to determine the fused coordinates as the acoustic positioning coordinates when the reliability and accuracy of the visual positioning coordinates are greatly reduced and the reliability and accuracy of the acoustic positioning coordinates are more reliable.

[0124] In S3052, according to the fused coordinates, the in-vehicle position information of the target inside the vehicle is determined.

[0125] In one implementation, it can be determined that the in-vehicle position information of the target inside the vehicle includes the fused coordinates.

[0126] In another implementation, the historical trajectory information of the target inside the vehicle in the past time can be combined to correct the fused coordinates to obtain the corrected coordinates, and it is determined that the in-vehicle position information of the target inside the vehicle includes the corrected coordinates, thereby improving the accuracy of the in-vehicle position information of the target inside the vehicle.

[0127] In one example, S3052 includes: in the case of a conflict between the visual positioning coordinates and the acoustic positioning coordinates, the fused coordinates are corrected according to the historical trajectory information of the target inside the vehicle in the past time to obtain the corrected coordinates, and the in-vehicle position information includes the corrected coordinates.

[0128] In this example, in the case where the visual positioning coordinates conflict with the acoustic positioning coordinates, the accuracy of the fused coordinates obtained based on the visual positioning coordinates and the acoustic positioning coordinates may be lacking. The fused coordinates can be corrected according to the historical trajectory information of the in-vehicle target in the past time to obtain corrected coordinates, making the movement trajectory of the in-vehicle target smoother and improving the accuracy of in-vehicle target positioning.

[0129] Optionally, determine whether the visual positioning coordinates conflict with the acoustic positioning coordinates according to the Euclidean distance between the visual positioning coordinates and the acoustic positioning coordinates. If the Euclidean distance between the visual positioning coordinates and the acoustic positioning coordinates is greater than the distance threshold, it can be determined that the visual positioning coordinates conflict with the acoustic positioning coordinates; otherwise, it is determined that the visual positioning coordinates and the acoustic positioning coordinates do not conflict. Thus, the accuracy of judging whether the visual positioning coordinates conflict with the acoustic positioning coordinates is improved.

[0130] Optionally, correct the fused coordinates according to the historical trajectory information of the in-vehicle target in the past time to obtain corrected coordinates, including: The iterative closest point (ICP) algorithm and the historical trajectory information of the in-vehicle target in the past time can be used to correct the fused coordinates to obtain corrected coordinates. On the one hand, the strong geometric constraint characteristics of the ICP algorithm are suitable for solving the mutation error caused by the instantaneous error of the sensor, which can improve the correction effect of the fused coordinates; on the other hand, the ICP algorithm does not require pre-training of the model and is suitable for in-vehicle real-time systems.

[0131] In this optional method, in the ICP algorithm, first, determine the coordinate point corresponding to the fused coordinates as the original coordinates. Then, in the iterative process, perform a nearest neighbor search on the original coordinates and the historical trajectory information to obtain a matching point pair, where the matching point pair includes the original coordinates and the trajectory point in the historical trajectory information that is closest to the original coordinates; based on the objective function, predict the transformation matrix between the matching point pairs, where the transformation matrix between the matching point pairs can include the rotation matrix between the matching point pairs and the translation vector between the matching point pairs; after that, transform the original coordinates according to the transformation matrix to obtain transformed coordinates; if the current iteration meets the iteration end condition, determine the corrected coordinates as the transformed coordinates, otherwise update the original coordinates as the transformed coordinates and continue the next iteration.

[0132] For example, the fused coordinates are represented as Pfused = (xfused, yfused, zfused), the predicted rotation matrix is R, the translation vector is t, and the transformed coordinates are represented as Pnew = R × Pfused + t.

[0133] For example, the objective function is the function that minimizes the sum of the squares of the distances between the matching point pairs.

[0134] For example, the iteration end condition may include one or more of the number of iterations being greater than a threshold number, the distance between the transformed coordinates and the trajectory points in the matching point pairs being less than a distance threshold, and the amount of change determined based on the change matrix being less than a change amount threshold.

[0135] Optionally, if the degree of deviation between the corrected coordinates and the historical trajectory information is greater than a degree threshold, an abnormal prompt message may be output so that the driver can promptly notice the abnormal position of the in-vehicle target.

[0136] In addition to the ICP algorithm, an extended kalman filter (EKF) algorithm, a particle filter algorithm, or a normal distributions transform (NDT) algorithm can also be used to correct the fused coordinates. In the extended kalman filter algorithm, the historical trajectory information of the in-vehicle target can be used as the state vector, and the fused coordinates can be used as the current observation data. Combining the state vector and the observation data, the corrected coordinates are predicted. This method has a low computational complexity and can improve the efficiency of correcting the fused coordinates. In the particle filter algorithm, a particle set can be used to represent the possible position distribution of the in-vehicle target, which is suitable for processing multi-modal data conflict scenarios. In the normal distributions transform algorithm, the historical trajectory information can be modeled as a probability density function, and by matching the probability density function with the fused coordinates, the corrected coordinates are determined. This method has stronger robustness.

[0137] In the embodiments of the present application, by combining the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, the visual positioning coordinates and the acoustic positioning coordinates are fused to obtain fused coordinates. Based on the fused coordinates, the in-vehicle position information of the in-vehicle target is finally obtained, effectively improving the positioning accuracy of the in-vehicle target by combining multi-modal data.

[0138] In some embodiments, after obtaining the in-vehicle position information of the in-vehicle target, the trajectory information of the in-vehicle target can be updated according to the in-vehicle position information of the in-vehicle target. Thus, the accuracy of the in-vehicle target trajectory is improved, and accurate tracking of the in-vehicle target is achieved based on the trajectory information of the in-vehicle target.

[0139] Exemplary device

[0140] Corresponding to the above in-vehicle target positioning method, the embodiments of the present application also provide an in-vehicle target positioning device. Figure 4 It is a schematic structural diagram of an in-vehicle target positioning device provided by the embodiments of the present application. As Figure 4 shown, the in-vehicle target positioning device 40 provided by the embodiments of the present application includes: a data acquisition unit 41, a spatio-temporal synchronization unit 42, a visual positioning unit 43, an acoustic positioning unit 44, and a position determination unit 45:

[0141] A data acquisition unit 41 acquires in-vehicle visual data and in-vehicle sound data; a spatio-temporal synchronization unit 42 is used to perform spatio-temporal synchronization on the in-vehicle visual data and the in-vehicle sound data; a visual positioning unit 43 is used to perform positioning of in-vehicle targets based on the spatio-temporally synchronized in-vehicle visual data to obtain a visual positioning result of the in-vehicle targets; an acoustic positioning unit 44 is used to perform positioning of in-vehicle targets based on the spatio-temporally synchronized in-vehicle sound data to obtain an acoustic positioning result of the in-vehicle targets; a position determination unit 45 is used to determine the in-vehicle position information of the in-vehicle targets according to the visual positioning result of the in-vehicle targets and the acoustic positioning result of the in-vehicle targets.

[0142] In some implementation manners, in the spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data, the time synchronization process of the in-vehicle visual data and the in-vehicle sound data includes: establishing a unified timestamp for the in-vehicle visual data and the in-vehicle sound data to obtain the in-vehicle visual data after time synchronization and the in-vehicle sound data after time synchronization.

[0143] In some implementation manners, in the spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data, the space synchronization process of the in-vehicle visual data and the in-vehicle sound data includes: converting the in-vehicle visual data after time synchronization from the visual sensor coordinate system to a set in-vehicle coordinate system to obtain the in-vehicle visual data after spatio-temporal synchronization; converting the in-vehicle sound data after time synchronization from the acoustic sensor coordinate system to the in-vehicle coordinate system to obtain the in-vehicle sound data after spatio-temporal synchronization.

[0144] In some implementation manners, the visual positioning unit 43 is specifically used for: performing target detection on the spatio-temporally synchronized in-vehicle visual data to obtain the visual positioning coordinates of the in-vehicle targets and the visual detection scores corresponding to the visual positioning coordinates; determining the confidence score corresponding to the visual positioning coordinates according to the optical radiation index and the visual detection scores of the in-vehicle visual data; wherein, the visual positioning result includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates.

[0145] In some implementation manners, the visual positioning unit 43 is specifically used for: weighting the optical radiation index and the visual detection scores according to the weight parameters corresponding to the optical radiation index and the weight parameters corresponding to the visual detection scores to obtain the confidence score corresponding to the visual positioning coordinates.

[0146] In some implementation manners, the acoustic positioning unit 44 is specifically used for: performing target positioning on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic positioning coordinates of the in-vehicle targets; determining the confidence score corresponding to the acoustic positioning coordinates according to the sound quality index of the in-vehicle sound data.

[0147] In some implementations, the voice quality metrics include the signal-to-noise ratio and the multipath interference metric. The acoustic positioning unit 44 is specifically configured to: weight the signal-to-noise ratio and the multipath interference metric according to the weight parameter corresponding to the signal-to-noise ratio and the weight parameter corresponding to the multipath interference metric, so as to obtain the confidence score corresponding to the acoustic positioning coordinates.

[0148] In some implementations, the visual positioning result includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates, and the acoustic positioning result includes the acoustic positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates. The position determination unit 45 is specifically configured to: fuse the visual positioning coordinates and the acoustic positioning coordinates according to the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, so as to obtain the fused coordinates corresponding to the in-vehicle target; determine the in-vehicle position information according to the fused coordinates.

[0149] In some implementations, the position determination unit 45 is specifically configured to: when the confidence score corresponding to the visual positioning coordinates is greater than the first score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the second score threshold, weight the visual positioning coordinates and the acoustic positioning coordinates according to the weight parameter corresponding to the visual positioning coordinates and the weight parameter corresponding to the acoustic positioning coordinates, so as to obtain the fused coordinates; when the confidence score corresponding to the visual positioning coordinates is less than the third score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the fourth score threshold, determine the fused coordinates as the acoustic positioning coordinates.

[0150] In some implementations, the position determination unit 45 is specifically configured to: when the visual positioning coordinates conflict with the acoustic positioning coordinates, correct the fused coordinates according to the historical trajectory information of the in-vehicle target in the past time, so as to obtain the corrected coordinates, and the in-vehicle position information includes the corrected coordinates.

[0151] In some implementations, the position determination unit 45 is specifically configured to: determine whether the visual positioning coordinates conflict with the acoustic positioning coordinates according to the Euclidean distance between the visual positioning coordinates and the acoustic positioning coordinates.

[0152] In some implementations, the in-vehicle target positioning device 40 further includes: a trajectory update unit (not shown in the figure), which updates the trajectory information of the in-vehicle target according to the in-vehicle position information.

[0153] The in-vehicle target positioning device provided in this embodiment belongs to the same inventive concept as the in-vehicle target positioning method provided in the above embodiments of the present application, and can execute the in-vehicle target positioning method provided in any of the above embodiments of the present application, and has the corresponding functional modules and beneficial effects for executing the in-vehicle target positioning method. For the technical details not described in detail in this embodiment, reference may be made to the specific processing content of the in-vehicle target positioning method provided in the above embodiments of the present application, which will not be elaborated here.

[0154] The functions implemented by the above data acquisition unit 41, spatio-temporal synchronization unit 42, visual positioning unit 43, acoustic positioning unit 44, and position determination unit 45 can be implemented by the same or different processors respectively, which is not limited in the embodiments of the present application.

[0155] It should be understood that the units in the above device can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or the functions of each unit of the device. The processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory inside or outside the device. Alternatively, the units in the device can be implemented in the form of a hardware circuit. By designing the hardware circuit, the functions of some or all of the units can be implemented. The hardware circuit can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationship of the components in the circuit. Again, for example, in another implementation, the hardware circuit can be implemented by a PLD. Taking an FPGA as an example, it can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured through a configuration file, so as to implement the functions of some or all of the above units. All the units of the above device can be implemented entirely in the form of a processor calling software, or entirely in the form of a hardware circuit, or partially in the form of a processor calling software, and the remaining part in the form of a hardware circuit.

[0156] In the embodiments of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor can be a circuit with the ability to read and execute instructions, such as a CPU, a microprocessor, a GPU, or a DSP, etc. In another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or can be reconstructed. For example, the processor is a hardware circuit implemented by an ASIC or a PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the configuration of the hardware circuit can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as a kind of ASIC, such as an NPU, a TPU, a DPU, etc.

[0157] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method. For example: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0158] In addition, all or part of the units in the above device can be integrated together or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of an SOC. The SOC may include at least one processor for implementing any of the above methods or the functions of each unit of the device. The types of the at least one processor can be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0159] Exemplary system

[0160] An embodiment of the present application provides a vehicle-mounted device. Refer to Figure 5 As shown, the vehicle-mounted device includes a memory 500 and a processor 510. Among them, the memory 500 is connected to the processor 510 and is used to store programs. The processor 510 is used to implement the in-vehicle target positioning method disclosed in any of the above embodiments by running the programs stored in the memory 500.

[0161] Specifically, the above vehicle-mounted device may further include: a bus, a communication interface 520, an input device 530, and an output device 540.

[0162] The processor 510, the memory 500, the communication interface 520, the input device 530, and the output device 540 are interconnected through the bus. Among them:

[0163] The bus may include a path for transmitting information between various components of the vehicle-mounted device.

[0164] The processor 510 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or may be an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present invention. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0165] The processor 510 may include a main processor and may also include a baseband chip, a modem, etc.

[0166] The program for implementing the technical solution of the present invention is stored in the memory 500, and the operating system and other key services can also be stored. Specifically, the program may include program codes, and the program codes include computer operation instructions. More specifically, the memory 500 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk memory, a flash memory, and so on.

[0167] The input device 530 may include devices for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor, etc.

[0168] The output device 540 may include devices for allowing information to be output to a user, such as a display screen, a printer, a speaker, etc.

[0169] The communication interface 520 may include devices of any transceiver type for communicating with other devices or communication networks, such as an Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0170] The processor 510 executes the program stored in the memory 500 and calls other devices, and can be used to implement each step of any one of the in-vehicle target positioning methods provided in the above embodiments of the present application.

[0171] An embodiment of the present application also proposes a chip, which includes a processor and a data interface. The processor reads and runs the program stored on the memory through the data interface to execute the in-vehicle target positioning method introduced in any of the above embodiments. The specific processing process and its beneficial effects can be referred to the embodiment introduction of the in-vehicle target positioning method above.

[0172] An embodiment of the present application also proposes a vehicle, which is equipped with the above-mentioned vehicle-mounted device for executing the steps of the above-mentioned in-vehicle target positioning method.

[0173] In one example, please refer to Figure 6 , the vehicle provided by the embodiment of the present application may include: a vision sensor 610, an acoustic sensor 620, a vehicle-mounted edge computing unit 630 (i.e., the vehicle-mounted device for executing the in-vehicle target positioning method), an output unit 640, a control unit 650, and a bus, and the bus is a vehicle CAN (controller area network) bus.

[0174] Among them, the visual sensor 610, the acoustic sensor 620, the vehicle-mounted edge computing unit 630, the output unit 640, and the control unit 650 perform data and / or signal transmission through the vehicle CAN bus.

[0175] Among them, the visual sensor 610 (such as an infrared camera array) is used for visual data acquisition; the acoustic sensor 620 (such as a distributed microphone array) is used for sound data acquisition. The vehicle-mounted edge computing unit 630 obtains the acquisition data of the visual sensor 610 and the acquisition data of the acoustic sensor 620 through the bus, and based on these acquisition data, executes each step of any one of the in-vehicle target positioning methods provided in the above embodiments of the present application. The vehicle-mounted edge computing unit 630 may refer to the vehicle-mounted device in the foregoing embodiments. The output unit 640 is used to output the in-vehicle position information and / or trajectory information of the in-vehicle target. The control unit 650 is used to perform corresponding control operations based on the in-vehicle sound data, such as turning on the air conditioner and adjusting the seat.

[0176] Exemplary computer program product and storage medium

[0177] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes computer program instructions. When the computer program instructions are run by a processor, the processor executes the steps in the in-vehicle target positioning method according to various embodiments of the present application described in any of the above embodiments of this specification.

[0178] The computer program product can be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present application. The programming languages include object-oriented programming languages such as Java and C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed completely on the user computing device, partially on the user device, executed as an independent software package, partially on the user computing device and partially on a remote computing device, or completely on a remote computing device or server.

[0179] In addition, the embodiments of the present application may also be a storage medium, on which a computer program is stored. The computer program is executed by a processor to perform the steps in the in-vehicle target positioning method according to various embodiments of the present application described in any of the above embodiments of this specification, and specifically can implement the steps of the in-vehicle target positioning method as described above.

[0180] For the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should understand that this application is not limited by the described action sequence, because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] It should be noted that the embodiments in this specification are all described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the similarities and differences between the embodiments, reference can be made to each other. For the device embodiments, since they are basically similar to the method embodiments, the description is relatively simple. For the relevant parts, reference can be made to the partial description of the method embodiments.

[0182] The steps in the method embodiments of this application can be adjusted, combined, and deleted according to actual needs. The technical features recorded in each embodiment can be replaced or combined.

[0183] The units of the devices in the embodiments of this application can be combined, divided, and deleted according to actual needs.

[0184] In the several embodiments provided by this application, it should be understood that the disclosed terminals, devices, and methods can be implemented in other ways. For example, the terminal embodiments described above are merely illustrative. For example, the division of modules or sub-modules is only a logical function division. In actual implementation, there can be other division methods. For example, multiple sub-modules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the displayed or discussed couplings or direct couplings or communication connections to each other can be through some interfaces. The indirect couplings or communication connections of devices or modules can be in electrical, mechanical, or other forms.

[0185] The modules or sub-modules described as separate components may or may not be physically separated. The components as modules or sub-modules may or may not be physical modules or sub-modules, that is, they can be located in one place, or they can be distributed to multiple network modules or sub-modules. Some or all of the modules or sub-modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, in each embodiment of this application, the functional modules or sub-modules can be integrated in a processing module, or each module or sub-module can exist physically alone, or two or more modules or sub-modules can be integrated in one module. The above-mentioned integrated modules or sub-modules can be implemented in the form of hardware or in the form of software functional modules or sub-modules.

[0187] Those skilled in the art may further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described according to functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0188] The steps of the methods or algorithms described in combination with the embodiments disclosed herein can be directly implemented by hardware, software units executed by a processor, or a combination of the two. The software units can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0189] Finally, it should also be noted that in this document, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover a non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0190] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to these embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for positioning an in-vehicle target, characterized in that, Including: Obtaining in-vehicle visual data and in-vehicle sound data; Performing spatio-temporal synchronization on the in-vehicle visual data and the in-vehicle sound data; Performing localization of in-vehicle targets based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual localization result of the in-vehicle targets; Performing localization of the in-vehicle targets based on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic localization result of the in-vehicle targets; Determining the in-vehicle position information of the in-vehicle targets according to the visual localization result and the acoustic localization result.

2. The in-vehicle target positioning method according to claim 1, characterized in that In the spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data, the time synchronization process of the in-vehicle visual data and the in-vehicle sound data includes: Establishing a unified timestamp for the in-vehicle visual data and the in-vehicle sound data to obtain the in-vehicle visual data after time synchronization and the in-vehicle sound data after time synchronization.

3. The in-vehicle target positioning method according to claim 2, wherein, In the spatio-temporal synchronization of the in-vehicle visual data and the in-vehicle sound data, the space synchronization process of the in-vehicle visual data and the in-vehicle sound data includes: Converting the in-vehicle visual data after time synchronization from the visual sensor coordinate system to a set in-vehicle coordinate system to obtain the in-vehicle visual data after spatio-temporal synchronization; Converting the in-vehicle sound data after time synchronization from the acoustic sensor coordinate system to the in-vehicle coordinate system to obtain the in-vehicle sound data after spatio-temporal synchronization.

4. The in-vehicle target positioning method according to any one of claims 1 to 3, characterized in that The performing localization of in-vehicle targets based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual localization result of the in-vehicle targets includes: Performing target detection on the spatio-temporally synchronized in-vehicle visual data to obtain the visual localization coordinates of the in-vehicle targets and the visual detection scores corresponding to the visual localization coordinates; Determining the confidence score corresponding to the visual localization coordinates according to the optical radiation index of the in-vehicle visual data and the visual detection scores; Wherein, the visual localization result includes the visual localization coordinates and the confidence score corresponding to the visual localization coordinates.

5. The in-vehicle target positioning method according to claim 4, characterized in that The determining the confidence score corresponding to the visual localization coordinates according to the optical radiation index of the in-vehicle visual data and the visual detection scores includes: Weighting the optical radiation index and the visual detection scores according to the weight parameter corresponding to the optical radiation index and the weight parameter corresponding to the visual detection scores to obtain the confidence score corresponding to the visual localization coordinates.

6. The in-vehicle target positioning method according to any one of claims 1 to 3, characterized in that, The performing localization of the in-vehicle targets based on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic localization result of the in-vehicle targets includes: Performing target localization on the spatio-temporally synchronized in-vehicle sound data to obtain the acoustic localization coordinates of the in-vehicle targets; Determining the confidence score corresponding to the acoustic localization coordinates according to the sound quality index of the in-vehicle sound data.

7. The in-vehicle target positioning method according to claim 6, characterized in that, The sound quality index includes a signal-to-noise ratio and a multipath interference index, and the determining the confidence score corresponding to the acoustic localization coordinates according to the sound quality index of the in-vehicle sound data includes: Based on the weight parameter corresponding to the signal-to-noise ratio and the weight parameter corresponding to the multipath interference index, the signal-to-noise ratio and the multipath interference index are weighted to obtain the confidence score corresponding to the acoustic positioning coordinates.

8. The in-vehicle target positioning method according to any one of claims 1 to 3, characterized in that, The visual positioning result includes the visual positioning coordinates and the confidence score corresponding to the visual positioning coordinates. The acoustic positioning result includes the acoustic positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates. Determining the in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result includes: Based on the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates, the visual positioning coordinates and the acoustic positioning coordinates are fused to obtain the fused coordinates corresponding to the in-vehicle target; Based on the fused coordinates, the in-vehicle position information is determined.

9. The in-vehicle target positioning method according to claim 8, wherein, The fusing the visual positioning coordinates and the acoustic positioning coordinates based on the confidence score corresponding to the visual positioning coordinates and the confidence score corresponding to the acoustic positioning coordinates to obtain the fused coordinates corresponding to the in-vehicle target includes: When the confidence score corresponding to the visual positioning coordinates is greater than the first score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the second score threshold, the visual positioning coordinates and the acoustic positioning coordinates are weighted based on the weight parameter corresponding to the visual positioning coordinates and the weight parameter corresponding to the acoustic positioning coordinates to obtain the fused coordinates; When the confidence score corresponding to the visual positioning coordinates is less than the third score threshold and the confidence score corresponding to the acoustic positioning coordinates is greater than the fourth score threshold, the fused coordinates are determined to be the acoustic positioning coordinates.

10. The in-vehicle target positioning method according to claim 8, wherein, The determining the in-vehicle position information based on the fused coordinates includes: When there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates, the fused coordinates are corrected according to the historical trajectory information of the in-vehicle target in the past time to obtain the corrected coordinates, and the in-vehicle position information includes the corrected coordinates.

11. The in-vehicle target positioning method according to claim 10, wherein Before the step of correcting the fused coordinates according to the trajectory information of the in-vehicle target in the past time to obtain the corrected coordinates when there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates, further includes: Based on the Euclidean distance between the visual positioning coordinates and the acoustic positioning coordinates, it is determined whether there is a conflict between the visual positioning coordinates and the acoustic positioning coordinates.

12. The in-vehicle target positioning method according to any one of claims 1 to 3, characterized in that, After determining the in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result, further includes: Based on the in-vehicle position information, the trajectory information of the in-vehicle target is updated.

13. An in-vehicle target positioning device, characterized in that, Includes: A data acquisition unit that acquires in-vehicle visual data and in-vehicle sound data; A spatio-temporal synchronization unit for spatio-temporally synchronizing the in-vehicle visual data and the in-vehicle sound data; A visual positioning unit for positioning the in-vehicle target based on the spatio-temporally synchronized in-vehicle visual data to obtain the visual positioning result of the in-vehicle target; An acoustic positioning unit, configured to perform positioning of the in-vehicle target based on the in-vehicle sound data after spatio-temporal synchronization, so as to obtain an acoustic positioning result of the in-vehicle target; A position determination unit, configured to determine in-vehicle position information of the in-vehicle target according to the visual positioning result and the acoustic positioning result.

14. An electronic device, characterized in that, Comprising a memory and a processor; The memory is connected to the processor and is used for storing programs; The processor is configured to implement the in-vehicle target positioning method according to any one of claims 1 to 12 by running the programs in the memory.

15. A computer program product, characterized in that, Comprising a computer program which, when executed by a processor, implements the in-vehicle target positioning method according to any one of claims 1 to 12.

Images