A method, apparatus, medium, and device for travel compliance detection

Abstract

The application provides a technical solution for travel compliance detection, which comprises the following steps: collecting audio PCM data in a vehicle for a preset time length in real time, processing the data to obtain corresponding mel-frequency cepstral coefficients and audio signal-to-noise ratio data, and obtaining audio feature vector data according to the mel-frequency cepstral coefficients; obtaining inertial measurement data in real time and processing the data to obtain road flatness data and vibration feature vector data; determining an audio coefficient based on the audio signal-to-noise ratio data and determining a vibration coefficient based on the road flatness data; determining a compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient and the vibration feature vector data, and determining travel compliance based on the score and a benchmark threshold value. The application can quickly and accurately monitor the number of passengers in a vehicle in real time without increasing the vehicle-mounted hardware and invading the privacy of passengers, thereby efficiently realizing travel compliance detection.

Description

Technical Field

[0001] This invention relates to the field of internet travel service technology, and in particular to a technology for detecting travel service compliance. Background Technology

[0002] In the services provided by internet-based ride-hailing service platforms (hereinafter referred to as "the Platform"), if an order is a private booking (e.g., a private booking in a ridesharing service or a non-shared ride in a ride-hailing service), the passenger in that order (one person or multiple passengers traveling together for the same private booking) exclusively occupies the vehicle seat for that order, and the driver should not pick up any other passengers. However, some drivers may be driven by economic interests and, in the actual fulfillment of such orders, illegally pick up other passengers (i.e., passengers not on private bookings) through offline channels or other platforms. This behavior not only increases driving safety hazards but also seriously damages the experience of passengers on private bookings and may lead to the Platform facing the risk of complaints and customer compensation claims.

[0003] Currently, the platform faces the following challenges in monitoring passengers picking up non-exclusive orders during the fulfillment of exclusive orders: 1. Traditional satellite positioning (GPS and / or BeiDou) can only track the vehicle's trajectory, but cannot detect changes in the number of passengers inside the vehicle in real time. If the driver does not deviate from the planned route but picks up passengers along the way, it is difficult to detect by tracking the vehicle's trajectory. 2. Monitoring using in-vehicle hardware such as cameras or specific sensors can detect changes in the number of passengers inside the vehicle in real time, but the hardware cost is high. 3. Traditional continuous recording methods can determine the number of people by sound, but they are prone to infringing on the privacy of drivers and passengers, and the data transmission consumes a lot of mobile data and battery power, with a high false alarm rate.

[0004] Therefore, how to monitor changes in the number of passengers in a vehicle in real time without relying on external in-vehicle hardware or infringing on personal privacy, in order to achieve compliance detection of travel fulfillment for exclusive orders, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In order to at least partially solve the above-mentioned technical problems, the purpose of this invention is to provide a method, apparatus, medium and equipment for detecting travel compliance.

[0006] According to one aspect of the present invention, a method for detecting travel compliance is provided, wherein the method includes: Audio PCM data of a preset duration is collected in real time inside the vehicle. The audio PCM data is processed to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data. Based on the Mel frequency cepstral coefficients, audio feature vector data is obtained. Real-time acquisition and processing of inertial measurement data yields road surface smoothness data and vibration feature vector data; Based on the audio signal-to-noise ratio data, the audio coefficient is determined, and based on the road surface smoothness data, the vibration coefficient is determined. Based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, a compliance confidence score is determined, and based on the compliance confidence score and a first benchmark threshold, the travel compliance is judged.

[0007] Optionally, the step of processing the audio PCM data to obtain the corresponding Mel-frequency cepstral coefficients and audio signal-to-noise ratio data includes: The audio PCM data is desensitized to obtain desensitized audio PCM data, and the desensitized audio PCM data is then segmented and windowed to obtain the corresponding segmented data. Based on the framed data, the Mel frequency cepstral coefficients and audio signal-to-noise ratio data are determined respectively.

[0008] Optionally, obtaining the audio feature vector data based on the Mel frequency cepstral coefficients includes: Obtain statistical data of the Mel frequency cepstral coefficients; The Mel frequency cepstral coefficients are input into a pre-trained lightweight audio feature extraction model to identify the breathing frequency data and conversation rounds data of the people in the vehicle. The statistical data, the breathing frequency data, and the conversation rounds data are concatenated to form audio feature vector data.

[0009] Optionally, the audio PCM data acquired each time is stored in a circular buffer and destroyed from the circular buffer after the corresponding Mel frequency cepstral coefficients are obtained.

[0010] Optionally, the real-time acquisition and processing of inertial measurement data to obtain road surface smoothness data and vibration feature vector data includes: Real-time acquisition of inertial measurement data, and determination of the vehicle's vertical vibration acceleration data based on the inertial measurement data; Calculate the short-time variance of the vibration acceleration data and use it as road surface smoothness data; The vibration acceleration data is subjected to a fast Fourier transform to obtain the corresponding spectrum data, and based on the spectrum data, the frequency value corresponding to the energy peak in the preset frequency range and the energy proportion in the preset frequency range are determined. Based on the frequency value and the reference frequency value, determine the frequency deviation and its rate of change; The frequency deviation and its rate of change, the energy ratio, and the short-time variance are concatenated to form vibration feature vector data.

[0011] Optionally, the determination of the reference frequency value includes: Calculate the average frequency value corresponding to the energy peak values ​​within all the preset frequency intervals determined within a preset time period from the start of acquiring inertial measurement data, and determine the average value as the reference frequency value.

[0012] Optionally, the determination of the reference frequency value includes: Based on the VIN code of the vehicle performing the contract, the reference frequency value is obtained from the vehicle model database.

[0013] Optionally, in step D, determining the audio coefficients based on the audio signal-to-noise ratio data includes: Based on the audio signal-to-noise ratio data, the audio coefficients are determined using the following formula:

[0014] in, Represents audio coefficients. As a preset constant, This represents the audio signal-to-noise ratio data. This represents the signal-to-noise ratio threshold.

[0015] Optionally, determining the vibration coefficient based on the road surface smoothness data includes: Based on the road surface smoothness data, the vibration coefficient is determined using the following formula:

[0016] in, Indicates the vibration coefficient. As a preset constant, This represents data on road surface smoothness. This indicates the threshold for road surface smoothness.

[0017] Optionally, the step of determining the compliance confidence score based on the audio coefficients, the audio feature vector data, the vibration coefficients, and the vibration feature vector data includes: The audio coefficient and the vibration coefficient are normalized respectively; The audio feature vector data and the vibration feature vector data are multiplied by their respective normalized coefficients and then input into a pre-trained lightweight fusion model to obtain the compliance confidence score.

[0018] Optionally, if either the audio feature vector data or the vibration feature vector data is unavailable, the compliance confidence score is determined based on the obtained audio feature vector data or vibration feature vector data and their corresponding normalization coefficients.

[0019] Optionally, the determination of the first benchmark threshold includes: The first benchmark threshold is determined based on the mean and standard deviation of all compliance confidence scores within a preset time period from the start of audio PCM data collection and inertial measurement data acquisition.

[0020] Optionally, the step of determining travel compliance based on the compliance confidence score and the first benchmark threshold includes: If the compliance confidence score exceeds the first benchmark threshold, the travel is deemed non-compliant.

[0021] Optionally, if the compliance confidence score exceeds the second benchmark threshold but does not exceed the first benchmark threshold, the method further includes: Acquire radio data from inside the vehicle during the trip fulfillment process; Based on the radio data, it is determined whether the travel agreement is compliant.

[0022] Optionally, the determination of the second reference threshold includes: Calculate the mean of all compliance confidence scores determined within a preset time period from the start of audio PCM data acquisition and inertial measurement data acquisition, and determine the mean as the second benchmark threshold.

[0023] Optionally, before determining the compliance confidence score, the method further includes: Real-time acquisition of context related to travel fulfillment orders, and determination of context feature vector data based on the context; The determination of the compliance confidence score includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

[0024] According to another aspect of the present invention, an apparatus for detecting travel compliance is provided, wherein the apparatus comprises: The first module is used to collect audio PCM data of a preset duration in the vehicle in real time, process the audio PCM data to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data, and obtain audio feature vector data based on the Mel frequency cepstral coefficients. The second module is used to acquire and process inertial measurement data in real time to obtain road surface smoothness data and vibration feature vector data. The fourth module is used to determine the audio coefficient based on the audio signal-to-noise ratio data, and to determine the vibration coefficient based on the road surface smoothness data; The fifth module is used to determine the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, and to determine the compliance of the travel based on the compliance confidence score and the first benchmark threshold.

[0025] Optionally, the device further includes: The third module is used to acquire context related to travel fulfillment orders in real time, and determine context feature vector data based on the context; In the fifth module, determining the confidence score for compliance includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

[0026] According to another aspect of the present invention, a computer-readable medium is provided, wherein computer-readable instructions are stored on the medium, which are executed by a processor to implement part or all of any of the above methods.

[0027] According to another aspect of the present invention, an apparatus for detecting travel compliance is provided, wherein the apparatus comprises: One or more processors; and a memory storing computer-readable instructions that, when executed, cause the processor to perform some or all of the operations described above.

[0028] Compared with existing technologies, the present invention provides a method, apparatus, medium, and device for detecting travel compliance. The method includes: real-time acquisition of audio PCM data of a preset duration inside the vehicle; processing the audio PCM data to obtain corresponding Mel-frequency cepstral coefficients and audio signal-to-noise ratio (SNR) data; and obtaining audio feature vector data based on the Mel-frequency cepstral coefficients; real-time acquisition and processing of inertial measurement data to obtain road surface smoothness data and vibration feature vector data; determining audio coefficients based on the audio SNR data and determining vibration coefficients based on the road surface smoothness data; determining a compliance confidence score based on the audio coefficients, the audio feature vector data, the vibration coefficients, and the vibration feature vector data; and determining travel compliance based on the compliance confidence score and a first benchmark threshold. This invention processes real-time in-vehicle audio PCM data to obtain audio feature vector data and audio coefficients, and processes real-time acquired inertial measurement data to obtain vibration feature vector data and vibration coefficients. Based on the audio coefficients, audio feature vector data, vibration coefficients, and vibration feature vector data, it determines the driver's compliance confidence score for fulfilling exclusive orders, thereby judging the driver's compliance with the exclusive order's travel obligations. This allows for rapid and accurate real-time monitoring of changes in the number of passengers in a vehicle without adding additional in-vehicle hardware or infringing on passenger privacy, thus efficiently detecting compliance issues related to exclusive orders. Attached Figure Description

[0029] Other features, objects, and technical effects of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 A schematic diagram of a method for detecting travel compliance according to one aspect of the present invention is shown. Figure 2 This diagram illustrates an exemplary multimodal data fusion method for exclusive order fulfillment compliance. Figure 3 This diagram illustrates an exemplary logic curve for determining the compliance of exclusive order fulfillment. Figure 4 A schematic diagram of an apparatus for detecting travel compliance according to another aspect of the present invention is shown; The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0030] The present invention will now be described in further detail with reference to the accompanying drawings.

[0031] In a typical configuration of various embodiments of the present invention, the method execution entity, each trusted party of the system and / or each module of the device may include one or more processors (CPU), input / output interfaces, network interfaces and memory.

[0032] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0033] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include non-transitory computer-readable media, such as modulated data signals and carrier waves.

[0034] Existing internet-based ride-hailing platforms face the following challenges in monitoring non-exclusive order passengers during the fulfillment of exclusive rides: 1. Traditional satellite positioning (GPS and / or BeiDou) can only track the vehicle's trajectory, failing to detect real-time changes in passenger numbers. If the driver picks up passengers along the planned route, it's difficult to detect this by tracking the vehicle's trajectory. 2. Monitoring using in-vehicle hardware such as cameras or specific sensors can detect real-time changes in passenger numbers, but the hardware is costly. 3. Traditional continuous recording methods, while able to determine the number of passengers by sound, easily infringe on driver and passenger privacy, and data transmission consumes significant mobile data and battery power, with a high false alarm rate.

[0035] This invention, based on an internet-based travel service platform, provides a technical solution for detecting travel fulfillment compliance. It enables real-time monitoring of changes in the number of passengers in a vehicle without relying on external in-vehicle hardware or infringing on personal privacy, thus efficiently detecting compliance for exclusive bookings. This significantly improves the robustness and accuracy of compliance detection during the execution of exclusive bookings.

[0036] To further illustrate the technical means adopted and the effects achieved by the present invention, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings and embodiments and / or optional embodiments.

[0037] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of relevant information and / or data involved in this invention shall comply with relevant laws, regulations, and standards, shall not violate public order and good morals, shall not harm the public interest, and necessary confidentiality measures may be taken as needed.

[0038] Figure 1 The diagram illustrates a method for travel compliance detection according to one aspect of the present invention, wherein one embodiment of the method includes: S101 collects audio PCM data of a preset duration inside the vehicle in real time, processes the audio PCM data to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data, and obtains audio feature vector data based on the Mel frequency cepstral coefficients. S102 acquires and processes inertial measurement data in real time to obtain road surface smoothness data and vibration feature vector data; S104 determines the audio coefficient based on the audio signal-to-noise ratio data, and determines the vibration coefficient based on the road surface smoothness data; S105 determines the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, and determines the compliance of the trip based on the compliance confidence score and the first benchmark threshold.

[0039] This invention provides a method for detecting travel fulfillment compliance based on an internet-based travel service platform 10. The internet-based travel service platform 10 may include a user terminal (or client) 100, a server terminal 200, and a driver terminal 300. The server terminal 200 includes one or more computer devices capable of data acquisition, processing, storage, and transmission. Both the user terminal 100 and the driver terminal 300 include smart terminal devices with the corresponding application (or app) of the internet-based travel service platform installed. Users submit exclusive travel requests to the server terminal 200 through the app on the user terminal 100, including at least information such as origin, destination, and travel time. After accepting the request, the user terminal 100 generates an exclusive order and publishes it to the driver terminal 300. When a driver accepts the order through the app on the driver terminal 300, the exclusive order becomes effective. When the driver arrives at the origin and the passenger boards the vehicle, the driver triggers the execution of the exclusive order through the driver terminal 300, initiating the travel fulfillment process. During the travel fulfillment process, travel fulfillment compliance can be detected through the user terminal 100 of the passenger who published the travel request. The computer equipment mentioned includes, but is not limited to, servers, network hosts, single network servers, or network server clusters, and the smart terminal devices include, but are not limited to, smartphones, smart wearable devices, and tablets. Here, the computer equipment and smart terminal devices are merely examples; other existing or future devices and / or resource platforms that are applicable to this invention should also be included within the scope of protection of this invention, and are hereby incorporated by reference.

[0040] When the app in the user terminal 100 detects that a user-posted exclusive order has been triggered for execution (i.e., the exclusive order has entered the "fulfillment" stage), the method of this embodiment is triggered. In step S101, during the order execution process, audio PCM (Pulse Code Modulation) data of a preset duration inside the vehicle can be collected in real time through the microphone or microphone array of the user terminal 100 based on a preset sampling frequency. For example, the preset duration can be 5 seconds or 10 seconds. The collected audio PCM data is processed by the relevant software in the user terminal 100. Instead of obtaining the corresponding voice content, the corresponding Mel-Frequency Cepstral Coefficients (MFCC) and audio signal-to-noise ratio data are extracted. Based on the extracted MFCC, the corresponding audio feature vector data is obtained. The MFCC simulates the audio features of human hearing.

[0041] Continuing in this embodiment, in step S102, inertial measurement data can be acquired in real time and processed through the inertial measurement unit (IMU) of the user terminal 100 to obtain road surface smoothness data and vibration feature vector data.

[0042] Continuing in this embodiment, in step S104, the user terminal 100 can determine the audio coefficient based on the audio signal-to-noise ratio data obtained in step S101, so as to dynamically adjust the obtained audio feature vector data and reduce the impact of the ambient noise inside the vehicle on the performance compliance confidence score. It can also determine the vibration coefficient based on the road surface smoothness data obtained in step S102, so as to dynamically adjust the vibration coefficient according to the road conditions (road surface smoothness).

[0043] Continuing in this embodiment, in step S105, the user terminal 100 can determine the compliance confidence score based on the audio coefficient and vibration coefficient obtained in step S104, the audio feature vector data obtained in step S101, and the vibration feature vector data obtained in step S102, and determine the compliance of the trip based on the obtained compliance confidence score and the first benchmark threshold.

[0044] In the above embodiments, no new hardware deployment is required. The user terminal 100 of passengers who issue exclusive orders dynamically collects and processes multimodal data such as audio and inertial data in real time to obtain audio feature vector data and vibration feature vector data. The feature vector data of each dimension is adjusted and compensated based on the dynamically obtained coefficients of each dimension. Then, based on the adjusted and compensated feature vector data of each dimension, the compliance confidence score of the driver in executing the exclusive order is determined. The compliance confidence score and a benchmark threshold can be used to determine the compliance of the vehicle in executing the exclusive order. This significantly improves the robustness and accuracy of the compliance detection in the execution of exclusive orders.

[0045] Optionally, in step S101, processing the audio PCM data to obtain the corresponding Mel-frequency cepstral coefficients and audio signal-to-noise ratio data includes: The audio PCM data is desensitized to obtain desensitized audio PCM data, and the desensitized audio PCM data is then segmented and windowed to obtain the corresponding segmented data. Based on the framed data, the Mel frequency cepstral coefficients and audio signal-to-noise ratio data are determined respectively.

[0046] Audio PCM data is uncompressed raw digital audio stream, directly recording the sampled values ​​of the sound waveform. To protect the privacy of drivers and passengers, in this optional embodiment, the user terminal 100 can first de-identify the collected audio PCM data using a deployed SDK (Software Development Kit). While preserving the usability of the audio, sensitive identity information (e.g., voiceprints of specific speakers, private conversation content, etc.) is removed or replaced, making the de-identified audio PCM data unrecoverable or unable to identify specific individuals. Before de-identifying the audio PCM data, VAD (Voice Activity Detection) processing can be performed to remove redundant segments without speech (i.e., silence), reducing the amount of subsequent data processing and improving its accuracy. Next, the anonymized audio PCM data is segmented and windowed to obtain the corresponding framed data. The short-time stationarity of the speech signal can be utilized to divide the long sequence into overlapping frames (i.e., short segments). To reduce spectral leakage that may be caused by truncation at both ends of the frames, each frame is multiplied by a window function to obtain the corresponding framed data. For example, the window function could be [missing information]. Further processing of the segmentation function yields the corresponding MFCC and audio signal-to-noise ratio data. Specifically, the framed data can be subjected to an FFT (Fast Fourier Transform) and the modulus squared to obtain the corresponding power spectrum. The time-domain signal is then converted to the frequency domain, filtered using a Mel filter bank to obtain the logarithmic energy, and finally subjected to a DCT (Discrete Cosine Transform) to obtain the MFCC corresponding to the audio PCM data. The corresponding audio signal-to-noise ratio (SNR) data can also be calculated after determining the noise power and signal power in the framed data. Existing technologies can be used to determine the noise power and signal power in the framed data, and to calculate the SNR based on the noise power and signal power. This will not be elaborated on here.

[0047] Optionally, in step S101, obtaining the audio feature vector data based on the Mel frequency cepstral coefficients includes: Obtain statistical data of the Mel frequency cepstral coefficients; The Mel frequency cepstral coefficients are input into a pre-trained lightweight audio feature extraction model to identify the breathing frequency data and conversation rounds data of the people in the vehicle. The statistical data, the breathing frequency data, and the conversation rounds data are concatenated to form audio feature vector data.

[0048] In this optional embodiment, the user terminal 100 can statistically analyze the mean and variance of the MFCC obtained within a unit of time up to the current time (e.g., 5 minutes or 10 minutes). The current MFCC statistics can reflect the change in the number of sound sources inside the vehicle. The MFCC within the current unit of time (e.g., 5 minutes or 10 minutes) is input into a pre-trained lightweight audio feature extraction model to identify the breathing frequency data of the people inside the vehicle (generally, the more people in the vehicle, the higher the breathing frequency) and the conversation rounds data (generally, the more people in the vehicle, the more conversation rounds). The obtained MFCC mean, variance, breathing frequency data, and conversation rounds data are then concatenated to form audio feature vector data. An exemplary audio feature vector data. It can be represented as follows:

[0049] in, The mean MFCC value per unit time; The variance of MFCC per unit time; The respiratory rate data of the people inside the vehicle within a unit of time is used for identification. This involves identifying the breathing frequency data of conversations among occupants within a vehicle per unit time. Specifically, the MFCC (Multiple Frequency Control) data per unit time can be input into a pre-trained lightweight audio feature extraction model to obtain... and The lightweight audio feature extraction model can be pre-trained using a cropped version of MobileNet-v2 or a custom 1D-CNN lightweight neural network. The number of parameters of the lightweight neural network is controlled at the level of hundreds of KB, which can be deployed on mainstream smart terminal platforms. In actual deployment, it can be adapted according to the relevant interfaces provided by the platform used by the terminal device (such as iOS CoreML, Android NNAPI, etc.).

[0050] Optionally, in step S101, the audio PCM data collected each time is stored in a circular buffer and destroyed from the circular buffer after the corresponding Mel frequency cepstral coefficients are obtained.

[0051] To ensure driver and passenger privacy, in this optional embodiment, in step S101, the user terminal 100 can preset a ring buffer of a certain storage size in memory to temporarily store audio PCM data of a preset duration collected each time. After processing the audio PCM data and obtaining the corresponding MFCC, the audio PCM data is destroyed from the ring buffer. The collected audio PCM data is not persistently stored; only the corresponding irreversible MFCC is retained for subsequent processing, thus protecting driver and passenger privacy from the source. The smaller the ring buffer, the lower the risk of driver and passenger privacy leakage. The storage space of the ring buffer only needs to be sufficient to store one audio PCM data of a preset duration. After obtaining the MFCC corresponding to the audio PCM data, the audio PCM data in the ring buffer can be destroyed before collecting the next audio PCM data. One method to destroy audio PCM data in the circular buffer is active deletion. This involves explicitly rewriting all positions in the circular buffer to zeros after MFCC extraction of the audio PCM data. This method destroys data promptly and is safer, but requires an additional zero-data write operation. Another method is to automatically overwrite the previous audio PCM data when the next acquired audio PCM data is written to the circular buffer. This method utilizes the inherent characteristic of the circular buffer (automatically overwriting old data with new data), incurring no additional overhead, but the old data still physically exists until it is overwritten. For example, the size of the circular buffer = sampling rate x bit depth x preset duration, such as 16kHz x 16bit x 5s = 160KB.

[0052] Optionally, in step S102, the real-time acquisition and processing of inertial measurement data to obtain road surface smoothness data and vibration feature vector data includes: Real-time acquisition of inertial measurement data, and determination of the vehicle's vertical vibration acceleration data based on the inertial measurement data; Calculate the short-time variance of the vibration acceleration data and use it as road surface smoothness data; The vibration acceleration data is subjected to a fast Fourier transform to obtain the corresponding spectrum data, and based on the spectrum data, the frequency value corresponding to the energy peak in the preset frequency range and the energy proportion in the preset frequency range are determined. Based on the frequency value and the reference frequency value, determine the frequency deviation and its rate of change; The frequency deviation and its rate of change, the energy ratio, and the short-time variance are concatenated to form vibration feature vector data.

[0053] In this optional embodiment, the user terminal 100 can acquire inertial measurement data (including acceleration, angular velocity, etc. in three-dimensional space) in real time through its built-in inertial measurement unit (IMU). Using the attitude fusion algorithm provided by the relevant SDK, the user terminal 100 terminal device's attitude inside the vehicle is calculated. Then, the coordinate system based on the terminal device is converted to the geodetic coordinate system (or the vehicle's geodetic coordinate system) through a rotation matrix. After passing a high-pass filter to filter out the DC component (i.e., the gravitational acceleration which can be regarded as a constant value), the net acceleration in the vertical direction is extracted. In the geodetic coordinate system, the net acceleration of the mobile phone in the vertical direction and the net acceleration of the vehicle in the vertical direction can be regarded as the same. Therefore, this net acceleration can be determined as the vibration acceleration data of the vehicle in the vertical direction. Among them, the attitude fusion algorithm can adopt complementary filtering (which utilizes the complementary characteristics of high and low frequencies, with the gyroscope in the IMU responsible for high-frequency dynamic attitude output and the accelerometer in the IMU responsible for low-frequency drift correction) or Kalman filtering (which completes the optimal estimation through a recursive process of "prediction-update", that is, first use the gyroscope data to predict the attitude at the next moment, and then use the accelerometer measurement value to correct the prediction result).

[0054] The user terminal 100 can also calculate the short-time variance of the obtained vibration acceleration data and use the obtained short-time variance as road surface smoothness data. The short-time variance refers to the statistical measure of the degree of local fluctuation of the signal within an extremely short time window (usually on the order of milliseconds, for example, 10–40 milliseconds), which is used to characterize the amplitude dispersion or frequency fluctuation intensity of non-stationary signals under the assumption of "short-time stationarity".

[0055] The user terminal 100 also performs FFT on the obtained vibration acceleration data to obtain the corresponding spectrum data, and determines the frequency value corresponding to the energy peak within a preset frequency range and the energy proportion within that preset frequency range. Then, based on this frequency value and the reference frequency value, it determines the frequency deviation and its rate of change. If the frequency value corresponding to the actual energy peak shifts to the left relative to the reference frequency value (the frequency value corresponding to the actual energy peak becomes smaller), it indicates an increase in the load inside the vehicle. Finally, the obtained frequency deviation and its rate of change, energy proportion, and short-time variance are concatenated to form vibration feature vector data. An example vibration feature vector data is provided. It can be represented as follows:

[0056] in, The frequency deviation between the frequency value corresponding to the energy peak value within a preset frequency range in the spectral data corresponding to vibration acceleration and the reference frequency value. The rate of change of frequency deviation; The frequency value corresponding to the energy peak value within a preset frequency range in the spectral data corresponding to vibration acceleration, and the energy percentage within that preset frequency range; Let be the short-time variance of the vibration acceleration.

[0057] Optionally, the determination of the reference frequency value includes: Based on the VIN code of the vehicle performing the contract, the reference frequency value is obtained from the vehicle model database.

[0058] Each vehicle has a unique Vehicle Identification Number (VIN), which drivers provide when registering on the internet ride-hailing service platform 10. The structure of each vehicle model is fixed, and the frequency value corresponding to the peak energy of its suspension under standard load within a preset frequency range (typically 1-5Hz) is a relatively fixed calibration value, which can be used as a reference frequency value. In this optional embodiment, the user terminal 100 can obtain the VIN of the vehicle accepting the order, and then determine the vehicle model based on the VIN. It can retrieve the reference frequency value corresponding to that model from a preset vehicle model database. For example, the platform's server 200 maintains a vehicle model database, and the user terminal 100 can access the server 200's vehicle model database to obtain the reference frequency value corresponding to the vehicle model executing the exclusive order.

[0059] Optionally, the determination of the reference frequency value includes: Calculate the average frequency value corresponding to the energy peak values ​​within all the preset frequency intervals determined within a preset time period from the start of acquiring inertial measurement data, and determine the average value as the reference frequency value.

[0060] The method of determining the reference frequency value through the VIN code reflects the standard load of the vehicle model. Different models may introduce errors, and the actual load should also consider the load of passengers in exclusive orders. To eliminate errors caused by different models and to consider the impact of the driver and passenger loads on the reference frequency value under compliant vehicle fulfillment, at the beginning of the fulfillment process (for example, when the status of the exclusive order in the user terminal 100 changes to "fulfilling," triggering real-time data collection, it can be considered that the exclusive order has started to be executed, and the time period from this point 3-5 minutes can be regarded as the beginning of the fulfillment process, hereinafter referred to as the reference stage), the driver generally will not pick up passengers who are not in the exclusive order. The passengers in the vehicle are the passengers of the exclusive order, and the frequency value corresponding to the energy peak value actually determined in this stage can be used as the reference frequency value. In this optional embodiment, the user terminal 100 can determine the corresponding vibration acceleration data based on the real-time acquired inertial measurement data, calculate the average of the frequency values ​​corresponding to all energy peak values ​​obtained in this stage within a preset frequency range (for example, 1-5Hz), and determine the average value as the reference frequency value.

[0061] Optionally, in step S104, determining the audio coefficients based on the audio signal-to-noise ratio data includes: Based on the audio signal-to-noise ratio data, the audio coefficients are determined using the following formula:

[0062] in, Represents audio coefficients. As a preset constant, This represents the audio signal-to-noise ratio data. This represents the signal-to-noise ratio threshold.

[0063] In this optional embodiment, the user terminal 100 can determine the audio coefficients using the following formula (1) based on the dynamically obtained audio signal-to-noise ratio data: (1) in, Indicates audio coefficients; These are constants that can be preset according to the application scenario; This represents the audio signal-to-noise ratio data dynamically obtained based on the frame data obtained after processing the acquired audio PCM data; Represents the signal-to-noise ratio threshold, where, It can be pre-calibrated through offline testing according to the vehicle model and application scenario. For example, a value of 15dB can be used (in general, in a car environment, when the audio signal-to-noise ratio is less than 10dB, the audio can be considered indistinguishable, and when it is greater than 20dB, the speech can be considered clear and the noise can be ignored). The sigmoid function can smoothly compress any number to between 0 and 1, and its function curve is S-shaped. According to the above formula (1), the noisier the car is, the smaller the SNR is. ,but The closer the signal-to-noise ratio is to 0, the more the contribution of the audio feature data vector to the confidence score of compliance can be dynamically adjusted based on the in-vehicle audio signal-to-noise ratio data.

[0064] Optionally, in step S104, determining the vibration coefficient based on the road surface smoothness data includes: Based on the road surface smoothness data, the vibration coefficient is determined using the following formula:

[0065] in, Indicates the vibration coefficient. As a preset constant, This represents data on road surface smoothness. This indicates the threshold for road surface smoothness.

[0066] In this optional embodiment, the user terminal 100 can determine the vibration coefficient based on the dynamically obtained road surface smoothness data using the following formula (2): (2) in, Indicates the vibration coefficient; These are constants that can be preset according to the application scenario; The road surface smoothness threshold can be represented by the short-time variance of the vertical vibration acceleration data of the same vehicle model obtained under the smooth road surface scenario obtained by pre-offline test calibration, or by the mean of the short-time variance of the vertical vibration acceleration data of the vehicle obtained in the baseline stage. The actual road surface smoothness data can be represented by the short-time variance of the vehicle's vertical vibration acceleration data obtained in real time. According to the above formula (2), the worse the road condition (the bumpier it is), the smoother the road surface. The smaller ,but The closer the value is to 0, the more the contribution of vibration feature vector data to the confidence score of performance compliance can be dynamically adjusted based on road surface smoothness data.

[0067] Optionally, in step S105, determining the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data includes: The audio coefficient and the vibration coefficient are normalized respectively; The audio feature vector data and the vibration feature vector data are multiplied by their respective normalized coefficients and then input into a pre-trained lightweight fusion model to obtain the compliance confidence score.

[0068] In this optional embodiment, in step S105, the user terminal 100 can first normalize the obtained audio coefficients and vibration coefficients respectively to obtain normalized audio coefficients and normalized vibration coefficients, which can be used as weights for the corresponding audio feature vector data and vibration feature vector data respectively. The normalized audio coefficients are multiplied with the corresponding audio feature vector data as weights, and the normalized vibration coefficients are multiplied with the vibration feature vector data as weights. The weighted audio feature vector data and weighted vibration feature vector data are then input into the pre-trained lightweight fusion model to obtain the compliance confidence score. For example, the following formulas (3) and (4) can be used to normalize the audio coefficients respectively. and vibration coefficient Normalization is performed: (3) (4) A lightweight multilayer perceptron neural network can be pre-trained to obtain a pre-trained lightweight fusion model. This lightweight multilayer perceptron neural network can include an input layer, multiple hidden layers, and an output layer, with the number of parameters controlled at the hundred KB level. It can be deployed on mainstream smart terminal platforms and can complete a single inference with a latency of less than 50 milliseconds on the input weighted audio feature vector data and weighted vibration feature vector data, outputting a compliance confidence score. In actual deployment, it can be adapted according to the relevant interfaces provided by the platform used by the terminal device (e.g., iOS CoreML, Android NNAPI, etc.).

[0069] Optionally, if either the audio feature vector data or the vibration feature vector data is unavailable, the compliance confidence score is determined based on the obtained audio feature vector data or vibration feature vector data and their corresponding normalization coefficients.

[0070] In this optional embodiment, if the user terminal device 100 is unable to obtain feature vector data of a certain mode of audio or vibration in real time for some reason, but can successfully obtain feature vector data of another mode—for example, it cannot collect audio PCM data, or although it can collect it, it cannot process it normally, resulting in the inability to obtain audio feature vector data; or it cannot acquire inertial measurement data, or although it acquires it normally, it cannot process it normally, resulting in the inability to obtain vibration feature vector data—then a downgraded processing method can be adopted. Only the obtained feature vector data of a single mode, i.e., audio feature vector data or vibration feature vector data, is input into a pre-trained lightweight fusion model that can dynamically adapt to the input to obtain a compliance confidence score.

[0071] Optionally, in step S105, determining the first benchmark threshold includes: The first benchmark threshold is determined based on the mean and standard deviation of all compliance confidence scores within a preset time period from the start of audio PCM data collection and inertial measurement data acquisition.

[0072] During the baseline phase (i.e., the preset time length from the start of audio PCM data acquisition and inertial measurement data acquisition), the passengers in the vehicle are passengers with exclusive orders, and there are no passengers with non-exclusive orders. In this phase, the audio feature vector data and vibration feature vector data obtained from the real-time acquired audio PCM data correspond to the driver and passenger situation under compliant performance. In this optional embodiment, the user terminal 100 can calculate the mean and standard deviation of all compliance confidence scores determined during the baseline phase to determine a first baseline threshold. For example, assuming the mean and standard deviation of all compliance confidence scores determined during the exclusive order execution phase are respectively... And s, the first benchmark threshold is The following formula (5) can be used to determine it. : (5) Here, k is the sensitivity coefficient. Decreasing the value of k increases sensitivity but also increases false alarms, while increasing the value of k decreases sensitivity but reduces false alarms. Typically, k is set to 3 based on the 3σ principle.

[0073] Optionally, in step S105, determining the travel compliance based on the compliance confidence score and the first benchmark threshold includes: If the compliance confidence score exceeds the first benchmark threshold, the travel is deemed non-compliant.

[0074] In this optional embodiment, in step S105, the user terminal 100 can compare the performance compliance confidence score output by the pre-trained lightweight fusion model with the obtained first benchmark threshold. If the performance compliance confidence score exceeds the first benchmark threshold, it is determined that the driver has committed a non-compliance in the process of executing the exclusive order.

[0075] Optionally, if the compliance confidence score exceeds the second benchmark threshold but does not exceed the first benchmark threshold, the method further includes: Acquire radio data from inside the vehicle during the trip fulfillment process; Based on the radio data, it is determined whether the travel agreement is compliant.

[0076] In this optional embodiment, the user terminal 100 can compare the performance compliance confidence score output by the pre-trained lightweight fusion model with a first benchmark threshold. If the performance compliance confidence score does not exceed the first benchmark threshold but exceeds the second benchmark threshold, the driver may have picked up passengers with non-exclusive orders during the performance process (potentially non-compliant performance). In this case, other modal data can be detected. For example, the user terminal 100 can acquire in-vehicle radio data during the trip and use the acquired radio data to help determine whether the trip performance is compliant. The radio data may include data obtained by detecting radio signals such as Bluetooth, WiFi, and / or UWB (Ultra Wide Band). For example, when the obtained compliance confidence score cannot directly determine whether the trip is compliant, the user terminal 100 can detect whether there are changes in the number of Bluetooth devices, WiFi addresses, or UWB devices. If the number of Bluetooth devices, WiFi addresses, or UWB devices increases, it can be determined that relevant devices have been added to the vehicle, which can serve as very strong evidence (basically confirming that there are new people in the vehicle), and can be used to determine that the trip is non-compliant. However, it cannot be used as a necessary condition (not detecting new devices does not mean that there are no new people in the vehicle - the new people may not have carried or turned off their devices).

[0077] However, the effectiveness of Bluetooth, WiFi, and / or UWB radio signal scanning is affected by whether the relevant modules / functions of the device are enabled (if the relevant functions are not enabled, they cannot be detected). Furthermore, even if enabled, devices belonging to people outside the vehicle may still be detected. Therefore, relying solely on Bluetooth, WiFi, and / or UWB radio signal scanning by the user terminal 100 to detect changes in the number of devices inside the vehicle to determine if new occupants have been added carries a certain possibility of missed or false detections, resulting in low reliability. When the obtained compliance confidence score cannot directly determine whether the travel compliance is in compliance, the user terminal 100 can acquire the radio data obtained during the Bluetooth, WiFi, or UWB radio signal scanning process and further process it to obtain radio statistical feature vector data. This radio statistical feature vector data can then be used to assist in determining whether the travel compliance is in compliance. Another example is to assume that the radio statistical feature vector data corresponding to the radio data obtained through Bluetooth, WiFi, or UWB radio signal scanning is... , can be represented as follows:

[0078] in, This indicates the number of newly added visible BLE broadcast devices relative to the baseline phase; This indicates the change in the number of WiFi request MAC addresses (even if the MAC addresses are random, the number of requests will increase as the number of devices increases); This indicates the duration of the newly added BLE or WiFi signal corresponding to the number of new additions (distinguishing between a brief passage by someone outside the vehicle and a new person inside, e.g., 30 seconds or 60 seconds). When the obtained compliance confidence score cannot directly determine whether the travel compliance is compliant, the obtained radio statistical characteristics can be used as a reference. After normalization, the corresponding auxiliary score is determined using the following formula (6). : (6) in, , , They are respectively for , , After normalization, b0 is the bias constant term, b1-b3 are the weights of each term, and the sum of the weights is 1.

[0079] For example, the following formulas (7), (8), and (9) can be used respectively to... , , Normalization is performed: (6) (7) (8) Where N is the maximum number of newly added Bluetooth devices. For example, if it is set to 5, and the number of newly added Bluetooth devices in the car exceeds this value... The value is 1; R is the upper limit of the frequency of WiFi address changes detected by WiFi. For example, if it is set to 10 times / minute, and the frequency of WiFi address changes in the car exceeds this value, then... The value is 1; T is the upper limit of the signal duration. For example, if it is set to 60 seconds, and a new Bluetooth device or WiFi address is added to the vehicle, its signal duration will exceed this value. The value is 1.

[0080] Will Compared with the baseline auxiliary score If a comparison is made, Greater than If the travel agreement is not fulfilled, a baseline auxiliary score can be set. This is a constant (for example, set to 0.6, meaning that when the obtained compliance confidence score cannot directly determine whether the travel compliance is compliant, an auxiliary score is used). If the value exceeds 0.6, the travel fulfillment is deemed non-compliant. Alternatively, radio statistical characteristics obtained from the baseline phase can be used. Determine the baseline auxiliary score according to the method in this example. .

[0081] It can also simplify the auxiliary judgment process. When the obtained performance compliance confidence score cannot directly determine whether the travel performance is compliant, the obtained radio statistical characteristics can be used instead. Directly inspect it. , and These values ​​are used to assist in the determination. For example, if... Not less than the preset quantity (e.g., 2) and Exceeding a preset time threshold (e.g., 30 seconds), or Exceeding a preset threshold (e.g., 5 times / minute) and If the travel time exceeds a preset time threshold (e.g., 30 seconds), it will directly assist in determining that the travel fulfillment is non-compliant; otherwise, the travel fulfillment is compliant.

[0082] The above are merely examples and are not intended to limit the scope of this invention. Other existing methods / algorithms that use radio data to assist in determining whether a travel obligation is compliant should be included within the scope of protection of this invention if applicable to this invention.

[0083] Optionally, the determination of the second reference threshold includes: Calculate the mean of all compliance confidence scores determined within a preset time period from the start of audio PCM data acquisition and inertial measurement data acquisition, and determine the mean as the second benchmark threshold.

[0084] During the baseline phase (i.e., the preset time length from the start of audio PCM data acquisition and inertial measurement data acquisition), the passengers in the vehicle are exclusive order passengers, and there are no non-exclusive order passengers with drivers. In this phase, the audio feature vector data and vibration feature vector data obtained from the real-time collected audio PCM data correspond to the driver-passenger situation under compliant performance. In this optional embodiment, the user terminal 100 can use the average of all compliance confidence scores determined during the baseline phase as a second baseline threshold. For example, assuming the average of all compliance confidence scores determined during the exclusive order execution phase is... Then you can As a second benchmark threshold.

[0085] Optionally, in step S105, before determining the compliance confidence score, the method for detecting travel compliance further includes: S103 ( Figure 1 (Not shown) Real-time acquisition of context related to travel fulfillment orders, and determination of context feature vector data based on the context; In step S105, determining the performance compliance confidence score includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

[0086] In this optional embodiment, the user terminal 100 can also obtain data such as the driver's historical performance and the execution time of the exclusive order in real time, and extract context feature data to obtain context feature vector data. The context feature vector data can be used as modal feature data and incorporated into the evaluation of the performance compliance confidence score, which can improve the coverage and accuracy of the evaluation of the performance compliance confidence score. In step S105, the user terminal 100 can determine the performance compliance confidence score based on the audio coefficient and vibration coefficient obtained in step S104, the audio feature vector data obtained in step S101, the vibration feature vector data obtained in step S102, the context feature vector data obtained in step S103, and the preset context coefficient. Based on the obtained performance compliance confidence score and a first benchmark threshold, the user terminal 100 determines the compliance of the travel performance. An exemplary context feature vector data is provided. It can be represented as follows:

[0087] in, The order type is identified using a unique hot code; for example, a unique order can be 1, and a non-exclusive order can be 0. A normalized historical integrity score for the driver; This is the normalization coefficient for risk during the order fulfillment period; Normalized index of congestion on road sections for order fulfillment.

[0088] Optionally, in step S105, the user terminal 100 can first consider the preset context coefficient, and normalize the context coefficient, the obtained audio coefficient, and the vibration coefficient respectively to obtain the normalized context coefficient, audio coefficient, and vibration coefficient. These can be used as weights for the corresponding audio feature vector data and vibration feature vector data, respectively. The normalized audio coefficient is multiplied with the corresponding audio feature vector data as a weight, the normalized vibration coefficient is multiplied with the vibration feature vector data as a weight, and the normalized context coefficient is multiplied with the context feature vector data as a weight. Then, the obtained weighted audio feature vector data, weighted vibration feature vector data, and weighted context feature vector data are input into the pre-trained lightweight fusion model to obtain the compliance confidence score. For example, assuming the preset context coefficient is... Formulas (10), (11), and (12) can be used to evaluate the context coefficients. Audio coefficients and vibration coefficient Normalization is performed: (10) (11) (12) The weighted context feature vector data, weighted audio feature vector data, and weighted vibration feature vector data can be input into a pre-trained lightweight fusion model to obtain a compliance confidence score.

[0089] The above embodiments and / or optional embodiments provide a method for detecting travel fulfillment compliance. Without requiring additional hardware, it can perform dual-modal (audio + vibration) detection of driver compliance when executing exclusive orders on the user's end (100-degree side), significantly reducing the false alarm rate of single-audio modal detection and improving accuracy. Optionally, the collected audio PCM data is temporarily stored in a circular buffer and not uploaded to the platform server. It is destroyed after obtaining the corresponding MFCC, maximizing driver and passenger privacy and completely eliminating concerns about privacy leaks at the technical level. Optionally, dynamically determining the benchmark threshold during the benchmark stage can eliminate the risk that the benchmark threshold determined based on the vehicle's VIN code may not match the actual situation (this method does not take driver and passenger load into consideration). Optionally, a dynamic weighting mechanism is employed. Based on dynamically acquired audio signal-to-noise ratio data and road surface smoothness data, the weights of audio feature vector data and vibration feature vector data are determined respectively. If the audio is heavily interfered with (e.g., high ambient noise inside the vehicle), the audio weight is automatically reduced and the vibration weight is increased. Conversely, if road conditions are poor (e.g., uneven or bumpy road surface), the vibration weight is automatically reduced and the audio weight is increased. Regardless of the environmental conditions, high-accuracy compliance confidence scores can be obtained through the complementarity of different modalities, significantly improving robustness. Furthermore, if the obtained compliance confidence score falls within the doubtful range, radio data can be dynamically obtained to assist in determining compliance. Additionally, contextual feature vector data reflecting factors such as the driver's historical compliance behavior and order execution time can be obtained as a modal feature vector data and incorporated into the evaluation of compliance confidence scores, further improving accuracy. On the other hand, passengers who book exclusive orders do not need to trigger them manually. Instead, they can trigger them seamlessly through the user's end. This "zero-interaction, zero-perception" method of travel fulfillment compliance detection maintains the natural atmosphere of social travel, avoids the damage to the trust relationship between drivers and passengers caused by strong intervention methods, and improves the user experience for both drivers and passengers.

[0090] An exemplary diagram illustrating multimodal data fusion for order fulfillment compliance is shown below. Figure 2 As shown, the user terminal 100 collects audio PCM data, inertial measurement data and context data in real time. After corresponding processing, the corresponding audio feature vector data, vibration feature vector data and context feature vector data are obtained, as well as their corresponding normalized coefficients. These are then input into a pre-trained fusion model to obtain the real-time compliance confidence score during the order fulfillment process.

[0091] An exemplary diagram illustrating the logic curve for determining order fulfillment compliance is shown below. Figure 3As shown, in the order scenario, during the entire journey from the start of fulfillment at time T0 to the end of fulfillment at time T4, T0 to T1 is the baseline stage, T1 to T2 is the fulfillment compliance stage, at time T2 the vehicle has additional load (which can be considered as a non-exclusive order passenger boarding), at time T3 the load is restored (which can be considered as a non-exclusive order passenger disembarking at the destination), T2 to T3 is the fulfillment non-compliance stage, and T3 to T4 is the fulfillment compliance stage.

[0092] The user-side (100) can also upload relevant data from the trip fulfillment process, such as real-time modal feature vector data and corresponding fulfillment compliance confidence scores and fulfillment compliance judgment results, to the platform's server (200). This data can serve as evidence for the platform to evaluate orders and assign responsibility (if applicable), which not only shortens the violation processing cycle but also significantly reduces customer complaint compensation expenses and manual review manpower input caused by driver non-compliance. Additionally, it can be used as historical data for iterative training of lightweight audio feature extraction and lightweight fusion models.

[0093] Figure 4 An apparatus for travel compliance detection according to another aspect of the present invention is shown, wherein, in one embodiment, the apparatus includes: The first module 410 is used to collect audio PCM data of a preset duration in the vehicle in real time, process the audio PCM data to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data, and obtain audio feature vector data based on the Mel frequency cepstral coefficients. The second module 420 is used to acquire and process inertial measurement data in real time to obtain road surface smoothness data and vibration feature vector data. The fourth module 440 is used to determine the audio coefficient based on the audio signal-to-noise ratio data, and to determine the vibration coefficient based on the road surface smoothness data; The fifth module 450 is used to determine the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, and to determine the compliance of the travel based on the compliance confidence score and the first benchmark threshold.

[0094] In this embodiment, the device is deployed in the user terminal of the Internet travel service platform, and its software and hardware environment is the same as that of the aforementioned user terminal 100.

[0095] In this embodiment of the device, the first module 410 of the device can acquire and process audio PCM data of a preset duration collected by the microphone or microphone array of the user terminal 100 inside the vehicle, extract the corresponding MFCC and audio signal-to-noise ratio data, and obtain the corresponding audio feature vector data based on the extracted MFCC.

[0096] Continuing with this device embodiment, the second module 420 of the device can acquire and process the inertial measurement data obtained by the IMU of the user terminal 100 based on the collected data to obtain road surface smoothness data and vibration feature vector data.

[0097] Continuing with this device embodiment, the fourth module 440 of the device can determine the audio coefficient based on the audio signal-to-noise ratio data obtained through the first module 410, so as to dynamically adjust the obtained audio feature vector data and reduce the impact of the ambient noise inside the vehicle on the performance compliance confidence score. It can also determine the vibration coefficient based on the road surface smoothness data obtained through the second module 420, so as to dynamically adjust the vibration coefficient according to the road conditions (road surface smoothness).

[0098] Continuing with this device embodiment, the fifth module 450 of the device can determine the compliance confidence score based on the audio coefficient and vibration coefficient obtained by the fourth module 440, the audio feature vector data obtained by the first module 410, and the vibration feature vector data obtained by the second module 420. Based on the obtained compliance confidence score and the first benchmark threshold, the compliance of the trip can be determined.

[0099] Optionally, the device for detecting travel compliance further includes: The third module 430 is used to acquire context related to travel fulfillment orders in real time, and determine context feature vector data based on the context; In the fifth module 450, the determination of the performance compliance confidence score includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

[0100] In this optional device embodiment, the third module 430 of the device can also acquire data such as the driver's historical performance and the execution time of exclusive orders in real time, extract contextual feature data, and obtain contextual feature vector data. The contextual feature vector data can be used as modal feature data and incorporated into the evaluation of performance compliance confidence score, which can improve the coverage and accuracy of the evaluation of performance compliance confidence score. Through the fifth module 450, the performance compliance confidence score can be determined based on the audio coefficient and vibration coefficient obtained by the fourth module 440, the audio feature vector data obtained by the first module 410, the vibration feature vector data obtained by the second module 420, the contextual feature vector data obtained by the third module 430, and the preset contextual coefficient. Based on the obtained performance compliance confidence score and the first benchmark threshold, the travel performance compliance is determined.

[0101] In the above-described device embodiments and / or optional embodiments, the functions / roles that can be executed / realized by the various component modules / parts of the device are the same as those in the aforementioned related method embodiments and / or optional embodiments, and will not be repeated here.

[0102] The device for detecting travel fulfillment compliance provided by the above-described embodiments and / or optional embodiments does not require additional hardware. It can perform dual-modal (audio + vibration) detection of driver fulfillment compliance when executing exclusive orders at the user end (100-degree end), significantly reducing the false alarm rate of single-audio modal detection and improving accuracy. Optionally, the collected audio PCM data is temporarily stored in a circular buffer and is not uploaded to the platform server. It is destroyed after obtaining the corresponding MFCC, which can maximize the protection of driver and passenger privacy. Optionally, the benchmark threshold is dynamically determined in the benchmark stage, which can eliminate the risk that the benchmark threshold determined based on the vehicle's VIN code may not match the actual situation (this method does not take driver and passenger load into consideration). Optionally, a dynamic weighting mechanism is employed. Based on dynamically acquired audio signal-to-noise ratio data and road surface smoothness data, the weights of audio feature vector data and vibration feature vector data are determined respectively. If the audio is heavily interfered with (e.g., high ambient noise inside the vehicle), the audio weight is automatically reduced and the vibration weight is increased. Conversely, if road conditions are poor (e.g., uneven or bumpy road surface), the vibration weight is automatically reduced and the audio weight is increased. Regardless of the environmental conditions, high-accuracy compliance confidence scores can be obtained through the complementarity of different modalities, significantly improving robustness. Furthermore, if the obtained compliance confidence score falls within the doubtful range, radio data can be dynamically obtained to assist in determining compliance. Additionally, contextual feature vector data reflecting factors such as the driver's historical compliance behavior and order execution time can be obtained as a modal feature vector data and incorporated into the evaluation of compliance confidence scores, further improving accuracy.

[0103] According to another aspect of the present invention, a computer-readable medium is also provided, the computer-readable medium storing computer-readable instructions that can be executed by a processor to implement some or all of the foregoing method embodiments and / or optional embodiments.

[0104] It should be noted that the method embodiments and / or optional embodiments in this invention do not strictly limit the order of execution of each step, as long as the method embodiments and / or optional embodiments can solve the defects existing in the prior art, achieve the inventive purpose of this invention, and obtain beneficial effects. The method embodiments and / or optional embodiments in this invention can be implemented in software and / or combinations of software and hardware. The software program involved in this invention can be executed by a processor to implement the steps or functions of the above embodiments. Similarly, the software program of this invention (including related data structures) can be stored in a computer-readable recording medium.

[0105] Furthermore, part or all of the present invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the present invention through the operation of the computer. The program instructions invoking the methods of the present invention may be stored in a fixed or removable recording medium, and / or transmitted via a data stream in a broadcast or other signal carrying medium, and / or stored in the working memory of a computer device operating according to the program instructions.

[0106] According to another aspect of the present invention, an apparatus for detecting travel compliance is also provided. The apparatus includes: a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the apparatus is triggered to run part or all of the methods and / or technical solutions of the foregoing embodiments.

[0107] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0108] In this invention, when terms such as "upper," "lower," "left," "right," "front," "rear," "top," "bottom," "inner," "outer," "middle," "vertical," "horizontal," "lateral," and "longitudinal" are used, the indicated orientation and / or positional relationship is based on the orientation and / or positional relationship shown in the accompanying drawings. These terms are primarily for the purpose of better describing this invention and its embodiments, and are not intended to limit the indicated device, element, or component to having a specific orientation, or to be constructed and operated in a specific orientation. Furthermore, some of the above terms, in addition to indicating orientation or positional relationship, can also be used to indicate other meanings; for example, the term "upper" can also be used in some cases to indicate a certain dependency or connection relationship. Those skilled in the art can understand the specific meaning of these terms in this invention according to the specific circumstances.

[0109] Furthermore, the terms "installation," "setup," "equipped with," "connection," and "linked" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral structure; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection via an intermediate medium; and they can refer to an internal connection between two devices, elements, or components. Those skilled in the art can understand the specific meaning of these terms in this invention based on the specific circumstances.

[0110] Furthermore, the terms "first," "second," etc., are primarily used to distinguish different devices, units, modules, elements, circuits, or components (which may be the same or different in specific type and construction), and are not intended to indicate or imply the relative importance, order, and / or quantity of the indicated devices, units, modules, elements, circuits, or components. Unless otherwise stated, "a plurality of" means two or more.

[0111] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device through software and / or hardware.

Claims

1. A method for detecting travel fulfillment compliance, characterized in that, The method includes: Audio PCM data of a preset duration is collected in real time inside the vehicle. The audio PCM data is processed to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data. Based on the Mel frequency cepstral coefficients, audio feature vector data is obtained. Real-time acquisition and processing of inertial measurement data yields road surface smoothness data and vibration feature vector data; Based on the audio signal-to-noise ratio data, the audio coefficient is determined, and based on the road surface smoothness data, the vibration coefficient is determined. Based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, a compliance confidence score is determined, and based on the compliance confidence score and a first benchmark threshold, the travel compliance is judged.

2. The method according to claim 1, characterized in that, The process of processing the audio PCM data to obtain the corresponding Mel-frequency cepstral coefficients and audio signal-to-noise ratio data includes: The audio PCM data is desensitized to obtain desensitized audio PCM data, and the desensitized audio PCM data is then segmented and windowed to obtain the corresponding segmented data. Based on the framed data, the Mel frequency cepstral coefficients and audio signal-to-noise ratio data are determined respectively.

3. The method according to claim 1, characterized in that, The step of obtaining audio feature vector data based on the Mel frequency cepstral coefficients includes: Obtain statistical data of the Mel frequency cepstral coefficients; The Mel frequency cepstral coefficients are input into a pre-trained lightweight audio feature extraction model to identify the breathing frequency data and conversation rounds data of the people in the vehicle. The statistical data, the breathing frequency data, and the conversation rounds data are concatenated to form audio feature vector data.

4. The method according to claim 1, characterized in that, Each acquired audio PCM data is stored in a circular buffer and destroyed from the circular buffer after the corresponding Mel frequency cepstral coefficients are obtained.

5. The method according to claim 1, characterized in that, The real-time acquisition and processing of inertial measurement data to obtain road surface smoothness data and vibration feature vector data includes: Real-time acquisition of inertial measurement data, and determination of the vehicle's vertical vibration acceleration data based on the inertial measurement data; Calculate the short-time variance of the vibration acceleration data and use it as road surface smoothness data; The vibration acceleration data is subjected to a fast Fourier transform to obtain the corresponding spectrum data, and based on the spectrum data, the frequency value corresponding to the energy peak in the preset frequency range and the energy proportion in the preset frequency range are determined. Based on the frequency value and the reference frequency value, determine the frequency deviation and its rate of change; The frequency deviation and its rate of change, the energy ratio, and the short-time variance are concatenated to form vibration feature vector data.

6. The method according to claim 5, characterized in that, The determination of the reference frequency value includes: Calculate the average frequency value corresponding to the energy peak values ​​within all the preset frequency intervals determined within a preset time period from the start of acquiring inertial measurement data, and determine the average value as the reference frequency value.

7. The method according to claim 5, characterized in that, The determination of the reference frequency value includes: Based on the VIN code of the vehicle performing the contract, the reference frequency value is obtained from the vehicle model database.

8. The method according to claim 1, characterized in that, The step of determining the audio coefficients based on the audio signal-to-noise ratio data includes: Based on the audio signal-to-noise ratio data, the audio coefficients are determined using the following formula: in, Represents audio coefficients. As a preset constant, This represents the audio signal-to-noise ratio data. This represents the signal-to-noise ratio threshold.

9. The method according to claim 1, characterized in that, The determination of the vibration coefficient based on the road surface smoothness data includes: Based on the road surface smoothness data, the vibration coefficient is determined using the following formula: in, Indicates the vibration coefficient. As a preset constant, This represents data on road surface smoothness. This indicates the threshold for road surface smoothness.

10. The method according to claim 1, characterized in that, The determination of the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data includes: The audio coefficient and the vibration coefficient are normalized respectively; The audio feature vector data and the vibration feature vector data are multiplied by their respective normalized coefficients and then input into a pre-trained lightweight fusion model to obtain the compliance confidence score.

11. The method according to claim 10, characterized in that, If either the audio feature vector data or the vibration feature vector data is unavailable, the compliance confidence score is determined based on the available audio feature vector data or vibration feature vector data and their corresponding normalization coefficients.

12. The method according to claim 1, characterized in that, The determination of the first benchmark threshold includes: The first benchmark threshold is determined based on the mean and standard deviation of all compliance confidence scores within a preset time period from the start of audio PCM data collection and inertial measurement data acquisition.

13. The method according to claim 1, characterized in that, The determination of travel compliance based on the compliance confidence score and the first benchmark threshold includes: If the compliance confidence score exceeds the first benchmark threshold, the travel is deemed non-compliant.

14. The method according to claim 13, characterized in that, If the compliance confidence score exceeds the second benchmark threshold but does not exceed the first benchmark threshold, the method further includes: Acquire radio data from inside the vehicle during the trip fulfillment process; Based on the radio data, it is determined whether the travel agreement is compliant.

15. The method according to claim 14, characterized in that, The determination of the second benchmark threshold includes: Calculate the mean of all compliance confidence scores determined within a preset time period from the start of audio PCM data acquisition and inertial measurement data acquisition, and determine the mean as the second benchmark threshold.

16. The method according to claim 1, characterized in that, Before determining the compliance confidence score, the method further includes: Real-time acquisition of context related to travel fulfillment orders, and determination of context feature vector data based on the context; The determination of the compliance confidence score includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

17. A device for detecting compliance with travel obligations, characterized in that, The device includes: The first module is used to collect audio PCM data of a preset duration in the vehicle in real time, process the audio PCM data to obtain the corresponding Mel frequency cepstral coefficients and audio signal-to-noise ratio data, and obtain audio feature vector data based on the Mel frequency cepstral coefficients. The second module is used to acquire and process inertial measurement data in real time to obtain road surface smoothness data and vibration feature vector data. The fourth module is used to determine the audio coefficient based on the audio signal-to-noise ratio data, and to determine the vibration coefficient based on the road surface smoothness data; The fifth module is used to determine the compliance confidence score based on the audio coefficient, the audio feature vector data, the vibration coefficient, and the vibration feature vector data, and to determine the compliance of the travel based on the compliance confidence score and the first benchmark threshold.

18. The apparatus according to claim 17, characterized in that, The device further includes: The third module is used to acquire context related to travel fulfillment orders in real time, and determine context feature vector data based on the context; In the fifth module, determining the compliance confidence score based on the audio coefficients, the audio feature vector data, the vibration coefficients, and the vibration feature vector data includes: Based on the audio coefficients, the audio feature vector data, the vibration coefficients, the vibration feature vector data, the context feature vector data, and the preset context coefficients, a compliance confidence score is determined.

19. A computer-readable medium, characterized in that, It stores computer-readable instructions that are executed by a processor to implement the method as described in any one of claims 1 to 16.

20. A device for detecting travel compliance, characterized in that, The device includes: One or more processors; and A memory storing computer-readable instructions, which, when executed, cause the processor to perform the method as described in any one of claims 1 to 16.

Images