Service recommendation method, device and computer readable storage medium

Abstract

The embodiment of the application relates to the technical field of data processing, and discloses a service recommendation method and device and a computer readable storage medium, the method comprising: acquiring multi-modal perception data of a vehicle; determining a target point of interest in a geographic space for an occupant based on the multi-modal perception data; inputting the multi-modal perception data into a multi-modal fusion network model to obtain an interest degree score of the target point of interest for the occupant; and recommending corresponding services to the occupant based on the interest degree score. The technical scheme of the application can reduce the deviation between the recommended services and the actual needs of the driver, and ensure that the recommended services are consistent with the actual intentions of the driver.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, specifically to a service recommendation method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the rapid development of intelligent vehicle technology, in-vehicle artificial intelligence systems are gradually shifting from passively executing commands to proactively predicting needs. In current in-vehicle interaction scenarios, users typically need to manually input information, with service recommendations based on this input. For example, searching for a destination on a navigation map and receiving a pre-planned navigation route; or, users can obtain services through simple voice commands, such as asking for directions to a nearby coffee shop. This approach not only easily distracts the driver but also fails to capture the driver's implicit needs, resulting in recommended services that do not match the driver's actual requirements. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a service recommendation method, apparatus and computer-readable storage medium to solve the problem in the prior art that the recommended services deviate from the actual needs of the driver due to the inability to accurately capture the driver's implicit needs.

[0004] According to one aspect of the present invention, a service recommendation method is provided, the method comprising: Acquire multimodal perception data of the vehicle; Based on multimodal perception data, determine the occupants' points of interest in geospatial targets; Multimodal sensing data is input into a multimodal fusion network model to obtain the occupant's interest rating of the target interest point; Based on interest ratings, corresponding services are recommended to passengers.

[0005] According to another aspect of the present invention, a service recommendation apparatus is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform the operation of the service recommendation method described above.

[0006] According to another aspect of the present invention, a computer-readable storage medium is provided, the storage medium storing at least one executable instruction, the executable instruction causing a service recommendation device to perform the following operations: acquiring multimodal perception data of a vehicle; determining the location of the occupant's gaze in geographic space based on the multimodal perception data, and obtaining the occupant's target point of interest in geographic space based on the location of the gaze; inputting the multimodal perception data into a multimodal fusion network model to obtain the occupant's interest score for the target point of interest; and recommending corresponding services to the occupant based on the interest score.

[0007] This invention first determines the occupant's target points of interest in geospatial space based on multimodal perception data, accurately identifying what the occupant is looking at. Then, a multimodal fusion network model comprehensively analyzes the multimodal perception data to determine the occupant's interest score for that target point of interest. This multimodal fusion network model can determine the occupant's interest in the target point of interest based on multiple dimensions of perception data, clarifying whether the target point of interest is the occupant's true point of interest, thereby improving the accuracy of target point of interest identification and precisely capturing the driver's implicit needs. Subsequently, based on the interest score of the target point of interest, corresponding services are recommended to the occupant, reducing the deviation between the recommended services and the driver's actual needs and ensuring that the recommended services match the driver's actual intentions.

[0008] The above description is merely an overview of the technical solutions of the embodiments of the present invention. In order to better understand the technical means of the embodiments of the present invention and to implement them in accordance with the contents of the specification, and to make the above and other objects, features and advantages of the embodiments of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0009] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a first embodiment of the service recommendation method provided by the present invention is shown; Figure 2 A detailed flowchart of step 120 of the service recommendation method provided by the present invention is shown; Figure 3 A detailed flowchart of step 130 of the service recommendation method provided by the present invention is shown; Figure 4 A schematic diagram of the model architecture of the multimodal fusion network model of the present invention is shown; Figure 5 A detailed flowchart of step 140 of the service recommendation method provided by the present invention is shown; Figure 6A flowchart illustrating a second embodiment of the service recommendation method provided by the present invention is shown. Figure 7 A flowchart illustrating the overall technical solution for the service recommendation of this invention is shown; Figure 8 A schematic diagram of the structure of a first embodiment of the service recommendation device provided by the present invention is shown. Detailed Implementation

[0010] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0011] With the rapid development of intelligent vehicle technology, in-vehicle artificial intelligence systems are gradually shifting from passively executing instructions to proactively predicting needs. In current in-vehicle interaction scenarios, users typically need to manually input information, such as searching for a destination on a navigation map, or use simple voice commands like "take me to the nearest coffee shop" to obtain services. This approach not only distracts the driver but also fails to capture the numerous implicit needs generated by users during the journey.

[0012] In existing technologies, some in-vehicle systems attempt to incorporate biometric technology to enhance the interactive experience. For example, in-car cameras capture the driver's facial orientation or gaze direction to determine if the driver is paying attention to a specific object on the roadside. However, most existing eye-tracking technologies are limited to "determining if the user is looking out the right window," making it impossible to pinpoint the user's specific point of interest. Furthermore, existing technologies often rely solely on single visual information such as head posture or gaze, lacking the fusion and analysis of multi-dimensional information such as speech semantics and physiological states like facial expressions. This makes it difficult for the system to accurately distinguish between a "casual glance" and genuine interest, easily leading to false triggers and impacting the user experience. Due to these shortcomings, existing technologies cannot accurately capture the driver's implicit needs, resulting in a discrepancy between recommended services and the driver's actual requirements.

[0013] To address the aforementioned shortcomings, this invention proposes a service recommendation method. The main technical solution includes: acquiring multimodal perception data of a vehicle; determining the occupant's target points of interest in the geographic space based on the multimodal perception data; inputting the multimodal perception data into a multimodal fusion network model to obtain the occupant's interest score for the target points of interest; and recommending corresponding services to the occupant based on the interest score.

[0014] This invention first determines the occupant's target points of interest in geospatial space based on multimodal perception data, accurately identifying what the occupant is looking at. Then, a multimodal fusion network model comprehensively analyzes the multimodal perception data to determine the occupant's interest score for that target point of interest. This multimodal fusion network model can determine the occupant's interest in the target point of interest based on multiple dimensions of perception data, clarifying whether the target point of interest is the occupant's true point of interest, thereby improving the accuracy of target point of interest identification and precisely capturing the driver's implicit needs. Subsequently, based on the interest score of the target point of interest, corresponding services are recommended to the occupant, reducing the deviation between the recommended services and the driver's actual needs and ensuring that the recommended services match the driver's actual intentions.

[0015] Figure 1 A flowchart of a first embodiment of the service recommendation method of the present invention is shown, which is executed by a service recommendation device. This service recommendation device can be an in-vehicle terminal, or other devices within the vehicle that have service recommendation functionality. Figure 1 As shown, the method includes the following steps: Step 110: Acquire multimodal perception data of the vehicle.

[0016] Multimodal perception data can be data collected through multiple channels, each with different data types. This multimodal perception data is used to subsequently determine the location of an occupant's gaze in geospatial space and the occupant's interest rating of a target point of interest. Using multimodal perception data, rather than a single data type, for subsequent analysis improves the accuracy of the determined gaze location and the accuracy of the occupant's interest rating of the target point of interest. The multimodal perception data here includes at least occupant data and vehicle data.

[0017] In one alternative approach, multimodal perception data of the vehicle can be acquired in real time. The acquired perception data for each modality is then aligned temporally and filtered. The average value of each filtered modality's perception data can be calculated to obtain more accurate multimodal perception data. The average value of each modality's perception data can be calculated using the above method, improving the accuracy of data acquisition across all dimensions. The calculated average value is then used as the acquired data for determining the interest score of the target interest point.

[0018] Step 120: Based on multimodal perception data, determine the occupant's points of interest in the geospatial target.

[0019] Points of interest (POIs) refer to geographical entities or locations within a geographic space that passengers visually focus on and develop an interest in, such as traffic signs, intersections, buildings, and landmarks.

[0020] In one alternative approach, the location of the occupant's line of sight in geographic space can be determined based on occupant and vehicle data. Finally, the location of the line of sight is matched spatially with a pre-stored database of points of interest (POIs), and the geographic points closest to the location of the line of sight and meeting spatial thresholds are selected as the occupant's target points of interest.

[0021] Step 130: Input the multimodal perception data into the multimodal fusion network model to obtain the passenger's interest rating of the target interest point.

[0022] The multimodal fusion network model is a deep learning neural network model that can receive different types of perceptual data, complete feature extraction, dimension alignment, cross-modal information fusion and inference computation of multimodal data, and realize joint modeling of multimodal perceptual data.

[0023] Interest score is a normalized value used to quantify the degree of attention or interest of passengers towards a target point of interest. It generally ranges from 0 to 1. The higher the interest score, the stronger the passenger's attention to the target point of interest, and the lower the interest score, the weaker the passenger's attention to the target point of interest.

[0024] In one alternative approach, the multimodal perception data is first standardized and preprocessed. For different modalities such as gaze data, speech data, and physiological data, feature vectors corresponding to each modality are extracted through fully connected layers and lightweight convolutional networks. Then, a feature alignment module is used to uniformly map features with inconsistent dimensions or scales to the same feature space, eliminating the differences between heterogeneous data. Subsequently, a multi-head attention mechanism is introduced to construct a multimodal fusion layer, which automatically learns the contribution weight of each modal perception data to the interest level. Finally, the fused features are input to the output layer to calculate and output the occupant's interest level score for the target interest point.

[0025] In another alternative approach, a multi-branch parallel network structure is constructed, with independent inference branches set for gaze data, speech data, and physiological data. Each branch extracts features from its corresponding modal perception data separately and outputs a preliminary single-modal interest score. A gated fusion unit is introduced to dynamically adjust the fusion weights of the preliminary single-modal interest scores of each branch, using the temporal changes in the occupant's gaze as a constraint. Then, a temporal smoothing layer is used to correct the fusion results of continuous time segments. Finally, the corrected fusion information is integrated to output a stable occupant interest score for the target point of interest.

[0026] Step 140: Recommend corresponding services to passengers based on interest scores.

[0027] In one optional approach, multi-level threshold ranges for interest ratings can be pre-defined, such as high interest, medium interest, and low interest. Simultaneously, a one-to-one mapping knowledge base between target points of interest and in-vehicle services can be established, such as matching landmarks with scenic spot explanations, shopping districts with restaurant and parking services, traffic intersections with traffic condition alerts, and scenic spots with navigation and tour services. The system reads the passenger's interest rating for the target points of interest in real time. If the rating is high, the system proactively pushes the complete service details corresponding to that point of interest and provides one-click navigation; if the rating is medium, only a simplified service prompt is pushed; if the rating is low, no service is pushed, thus achieving tiered and precise service recommendations.

[0028] Another optional approach involves continuously collecting passengers' continuous interest ratings for multiple target points of interest over a period of time, forming time-series interest data to uncover passengers' stable preference tendencies. Combining this with information such as the vehicle's real-time location, driving direction, and time context, related services that are spatially adjacent to and type-matched with the currently high-rated points of interest are selected. For example, if a passenger consistently gives high interest ratings to a business district, in addition to pushing basic services for that business district, additional related services such as popular nearby restaurants, parking lots, and leisure venues are recommended. Simultaneously, points of interest with lower ratings and their corresponding services are filtered to avoid redundant pushes, achieving personalized service recommendations that align with the passenger's long-term preferences.

[0029] In this embodiment, the location of the occupant's gaze in geographic space is first determined based on multimodal perception data to identify the occupant's target point of interest in geographic space, accurately identifying what the occupant is looking at. Subsequently, a multimodal fusion network model comprehensively analyzes the multimodal perception data to determine the occupant's interest score for that target point of interest. This multimodal fusion network model can determine the occupant's interest in the target point of interest based on multiple dimensions of perception data, clarifying whether the target point of interest is the occupant's true point of interest, thereby improving the accuracy of target point of interest identification and precisely capturing the driver's implicit needs. Following this, based on the interest score for the target point of interest, corresponding services are recommended to the occupant, reducing the deviation between the recommended services and the driver's actual needs and ensuring that the recommended services match the driver's actual intentions.

[0030] Reference Figure 2 In one optional approach, the multimodal perception data includes occupant data and vehicle data. The occupant data includes at least occupant line-of-sight data, and the vehicle data includes at least vehicle position information and vehicle attitude information. Based on the multimodal perception data, determining the occupant's points of interest in the geospatial target includes: Step 121: Based on line-of-sight data, vehicle location information, and vehicle attitude information, determine the location of the occupant's line of sight in geographic space.

[0031] The landing point is the occupant's line of sight, projected onto and intersecting with a geographic entity, such as a three-dimensional geographic coordinate point on the ground, building, or object surface. It represents the actual location of the line of sight in the real geographic environment. This landing point can be used to determine the occupant's point of interest in the geographic space.

[0032] The line-of-sight data can be obtained through cameras inside the vehicle. Specifically, the cameras can capture the rotation angle of the occupant's eyes, including the azimuth and pitch angles. The azimuth and pitch angles can be directly used as line-of-sight data, or the azimuth and pitch angles can be processed to obtain the line-of-sight direction, which can then be used as line-of-sight data.

[0033] Vehicle data includes at least vehicle position information, vehicle attitude information, and vehicle speed. This vehicle data can be collected through corresponding sensors, such as using BeiDou / GPS to collect vehicle position information, using an IMU (Inertial Measurement Unit) to collect the vehicle's heading / pitch angles to obtain vehicle attitude information, and using wheel speed sensors to collect vehicle speed. The vehicle position and attitude information in the vehicle data are used to subsequently determine the location of the occupants' line of sight in geographic space. By comprehensively analyzing multi-dimensional vehicle data, the accuracy of the location of the line of sight can be improved.

[0034] In one alternative approach, gaze data includes gaze direction and gaze depth. Gaze depth refers to the three-dimensional straight-line distance between the occupant's eye position along the gaze direction and the point of focus, representing the distance to the focus point. It is a depth parameter that characterizes the distance to the focus point and can be obtained through eye-tracking models, vehicle-mounted binocular depth camera ranging, monocular vision ranging, etc. It is used to constrain the spatial length of the gaze ray and is key depth information for calculating the focus point. The gaze direction can be converted into a three-dimensional direction vector in the vehicle coordinate system. Combined with the gaze depth, the relative coordinates of the occupant's gaze point in the vehicle coordinate system can be calculated. Using vehicle attitude information, an attitude rotation matrix is ​​constructed to correct the directional deviation caused by vehicle attitude tilt, completing the rotation transformation from the vehicle coordinate system to the geographic coordinate system. Finally, using the vehicle position information as the translation reference, the relative coordinates are translated geographically to obtain the focus point of the occupant's gaze in geographic space.

[0035] In another alternative approach, the line-of-sight data is first filtered and corrected to calculate the occupant's line-of-sight direction vector in the vehicle's body coordinate system. A rotation matrix is ​​then constructed using vehicle attitude information, and combined with vehicle position information, a translation and rotation transformation from the vehicle's body coordinate system to the geographic coordinate system is performed, converting the line-of-sight direction vector in the vehicle's body coordinate system into a three-dimensional line-of-sight ray in the geographic coordinate system. A digital elevation model and a three-dimensional real-scene geographic model are then imported to construct a geographic spatial entity surface. A spatial geometry algorithm is used to solve for the intersection point between the three-dimensional line-of-sight ray in the geographic coordinate system and the geographic spatial entity surface; this intersection point is the location where the line of sight lands in geographic space.

[0036] In another alternative approach, the geographic spatial area in front of the vehicle covered by the occupant's line of sight can be calculated based on vehicle location and attitude information, combined with gaze data. Simultaneously, the onboard real-view camera is used to capture real-view images of this geographic spatial area. A deep learning semantic segmentation model is then used to perform pixel-level recognition and classification of geographic entities such as roads, buildings, and signs within the real-view images. Through a coordinate mapping algorithm, the pixel coordinates of the real-view images are converted into three-dimensional geographic coordinates to determine the geographic landing point of the gaze in geographic space. Combining the gaze duration and the number of times the gaze is focused, the geographic entities identified around the landing point are weighted and filtered, and finally, the geographic entities that the occupant focuses on are extracted as target points of interest.

[0037] This process involves combining gaze duration and viewing frequency to weighted filtering of geographic entities identified around the gaze point, ultimately extracting entities of primary interest to the occupant. These entities include: using the calculated gaze point in geographic space as the center, a corresponding geographic space buffer radius is set according to different geographic scenarios (e.g., 10m for urban roads, 30m for highways); all geographic entities identified through real-world semantic recognition within this buffer radius are extracted to form a candidate geographic entity set. For each geographic entity in the candidate set, two types of behavioral data are collected: gaze duration (the total duration the gaze point falls within the geographic boundary of the entity) and viewing frequency (the number of times the occupant independently looks at the entity). Based on the characteristics of visual attention behavior, weighted coefficients for gaze duration and viewing frequency are set. For each candidate geographic entity in the set, a weighted sum is calculated using its corresponding gaze duration and viewing frequency, combined with the corresponding weighted coefficients, to obtain a comprehensive attention score for each candidate geographic entity. A preset attention score threshold is set, and geographic entities with a comprehensive attention score higher than the preset attention score threshold are identified as key candidates for the occupant's attention. If there are multiple candidate geographic entities, the geographic entity with the highest comprehensive attention score is selected as the final target point of interest for the occupant in the geographic space. If only a single geographic entity meets the conditions, that geographic entity is directly selected as the target point of interest for the occupant in the geographic space.

[0038] Based on the landing location, the occupants' points of interest in the geographic space include: Step 122: Match the landing location with the pre-stored point of interest database to obtain the occupant's target points of interest in the geographic space.

[0039] The pre-stored point-of-interest database is a structured geographic information dataset pre-stored on the vehicle terminal or in the cloud. It contains information such as the three-dimensional geographic coordinates, spatial boundary range, entity category, name, and attribute labels of various geographic target points of interest, covering geographic entities such as traffic signs, buildings, business districts, scenic spots, intersections, and road facilities. It is used to spatially match with the geographic landing point of the line of sight and identify the target objects that the occupant is interested in.

[0040] In one alternative approach, an adaptive spatial buffer radius is set based on the driving scenario, with the landing point as the center; the pre-stored interest point database is traversed to filter out all candidate interest points within the buffer range defined by the buffer radius; the spatial straight-line distance between the landing point and each candidate interest point is calculated, and the interest point with the closest spatial straight-line distance is selected as the occupant's target interest point.

[0041] In another optional approach, the spatial boundaries of each geographic entity in the pre-stored point of interest database are retrieved; it is determined whether the landing point falls within the spatial boundary of a geographic entity in the point of interest database; if so, the landing point of interest is taken as the target point of interest.

[0042] In this embodiment, by combining line-of-sight data, vehicle location information, and vehicle posture information, the location of the occupant's line of sight in geographic space is determined. This overcomes the deficiency that relying solely on the direction of the line of sight cannot determine the depth of gaze and is prone to causing ambiguity in the location of the location, thus ensuring the accuracy of the location. Furthermore, by matching the location of the location with a pre-stored database of points of interest, the system can accurately pinpoint the target points of interest that the occupant is actually paying attention to from a complex geographic environment. This provides a precise and reliable underlying perception basis for subsequently quantifying the occupant's interest and pushing personalized in-vehicle services.

[0043] Furthermore, in an alternative approach, step 121: determining the location of the occupant's line of sight in geographic space based on line-of-sight data, vehicle location information, and vehicle attitude information includes: Step 1211: Construct a line-of-sight vector in the occupant coordinate system based on the line-of-sight direction, and convert the line-of-sight vector into a first transformation vector in the vehicle coordinate system using a preset rotation matrix from occupant to vehicle.

[0044] In one alternative approach, a local coordinate system is established with the center point of the occupant's eyes as the origin. The horizontal and vertical deflection angles of the line of sight are converted into line of sight vectors in the occupant's coordinate system. A pre-calibrated and stored rotation matrix of the occupant relative to the vehicle is called. This rotation matrix is ​​used to compensate for the angular deviation between the occupant's sitting posture, head reference orientation, and the vehicle body. The line of sight vector in the occupant's coordinate system is multiplied by this rotation matrix to obtain the first transformation vector in the vehicle coordinate system.

[0045] Step 1212: Construct a rotation matrix from the vehicle coordinate system to the geographic coordinate system based on the vehicle attitude information, and use this rotation matrix to convert the first transformation vector into a second transformation vector in the geographic coordinate system.

[0046] In one alternative approach, vehicle attitude information, including heading angle, pitch angle, and roll angle, is extracted. A rotation matrix is ​​constructed to transform the vehicle coordinate system to the geographic coordinate system according to the Euler angle transformation rule. The first transformation vector in the vehicle coordinate system is multiplied by the rotation matrix to correct the line-of-sight direction shift caused by changes in vehicle roll, pitch, and driving orientation, thus obtaining the second transformation vector in the geographic coordinate system.

[0047] Step 1213: Determine the location of the occupant's line of sight in geographic space based on the vehicle location information, the second transformation vector, and the gaze depth.

[0048] In one alternative approach, the vehicle location information is parsed into reference coordinates in a geographic coordinate system. The second transformation vector is multiplied by the gaze depth to obtain the gaze space displacement vector. The reference coordinates and the gaze space displacement vector are then superimposed to calculate the final location of the occupant's gaze in geographic space.

[0049] In this embodiment, by progressively transforming the occupant coordinate system, vehicle coordinate system, and geographic coordinate system, the rotation matrix is ​​used to correct individual occupant posture deviations, and the rotation matrix constructed based on the vehicle posture compensates for vehicle motion posture interference. Combined with the gaze depth and vehicle position, the landing point is accurately calculated, realizing the mapping of gaze data to the real geographic space. This avoids the directional deviation and positioning error caused by direct coupling of multiple coordinate systems, improves the accuracy and stability of landing point position calculation, and provides a reliable foundation for subsequent matching of target interest points.

[0050] Reference Figure 3 In one optional approach, the multimodal perception data includes occupant data, which includes at least occupant gaze data, speech data, and physiological data; Step 130: Input the multimodal perception data into a multimodal fusion network model to obtain the occupant's interest rating for the target interest point, including: Step 131: Extract the features corresponding to gaze data, speech data, and physiological data from the input layer of the multimodal fusion network model.

[0051] Step 132: Align the features corresponding to gaze data, speech data, and physiological data through the feature alignment layer of the multimodal fusion network model.

[0052] Step 133: The aligned features are fused through the feature fusion layer of the multimodal fusion network model to obtain fused features. The fused features are then regressed and predicted by the multilayer perceptron (MLP) of the multimodal fusion network model to generate the original scores representing the intensity of passenger interest.

[0053] Step 134: Normalize the original scores through the output layer of the multimodal fusion network model to obtain the passenger's interest score for the target interest point.

[0054] Voice data can be collected through a microphone array inside the vehicle. Specifically, the voice data can be structured data generated after the microphone array collects the real-time voice signal of the occupants, performs noise reduction, speech recognition, and semantic parsing. This data includes keywords, semantics, tone, intonation, and emotional tendency, while excluding external environmental noise and in-vehicle equipment noise. It is used to uncover the explicit or implicit needs expressed by the occupants' subjective verbal expressions, serving as a semantic corroborating dimension for interest recognition and supplementing subjective intentions that cannot be recognized visually.

[0055] Physiological data can include occupant posture, heart rate, pressure on the steering wheel, and pressure on the seat, etc., which can be collected by various sensors. This physiological data can corroborate occupant interest levels from a bodily instinctual perspective, distinguishing between casual glances and active attention, and solving the problem of identifying genuine interest that cannot be determined by gaze and voice data.

[0056] Reference Figure 4 , Figure 4 This is a schematic diagram of the multimodal fusion network model of the present invention. A parallel feature extraction branch is built in the input layer of the multimodal fusion network model. For gaze data, a one-dimensional temporal convolutional network is used to extract temporal features such as gaze direction and gaze depth; for speech data, Mel-frequency cepstral coefficients are extracted, and emotional and semantic features are extracted through lightweight convolutional layers; for physiological data, heart rate features and occupant posture features are extracted through a single fully connected layer, and feature vectors corresponding to the three types of data are output respectively.

[0057] The feature alignment layer adopts adaptive dimensional mapping and standardization. Through a learnable mapping matrix, feature vectors with inconsistent dimensions and scales, such as gaze data, speech data, and physiological data, are uniformly mapped to a feature space of the same dimension. At the same time, Z-score standardization is performed on each feature to eliminate differences in scale and numerical bias, thus completing the alignment and matching of the three types of features.

[0058] The feature fusion layer adopts an attention-weighted concatenation method, which assigns dynamic weights to the three aligned features and then concatenates them to generate fused features. The fused features are then input into a multilayer perceptron (MLP) containing two hidden layers, and feature depth mining is completed through a nonlinear activation function to output the original scores representing the intensity of occupant interest.

[0059] The output layer uses the Sigmoid activation function to normalize the raw scores of the MLP output to the 0-1 range, converting the raw scores into interest scores with standardized value ranges, thus achieving standardized output of interest scores.

[0060] In this embodiment, multimodal differential features are extracted in parallel through the input layer to comprehensively explore the interest expression information of passengers at the visual, linguistic, and physiological levels, thus compensating for the one-sidedness of single-modal data. The feature alignment layer solves the problem of inconsistent dimensions and scales of heterogeneous data, ensuring that multimodal data can be effectively fused. Then, attention-weighted fusion and MLP nonlinear regression are used to explore the correlation features between multimodal data and accurately generate original scores. Finally, normalization is used to obtain the passenger's interest score for the target interest point, which effectively improves the accuracy, stability, and standardization of the interest score, and provides a reliable quantitative basis for subsequent accurate push of corresponding services based on the interest score.

[0061] Reference Figure 5 In one alternative approach, step 140: Based on interest ratings, recommending relevant services to passengers includes: Step 141: If the interest score is greater than or equal to the first preset score and less than the second preset score, load the relevant information of the target interest point and recommend the relevant information to the passengers.

[0062] The first preset score refers to a pre-set threshold, which is the minimum score required to trigger the push of basic information about the target point of interest. This first preset score can be fixed or dynamically adjusted based on vehicle speed. The second preset score refers to a pre-set threshold, which is higher than the first preset score, and is the score required to trigger destination-level travel services. This second preset score can also be fixed or dynamically adjusted based on vehicle speed.

[0063] The relevant information of the target point of interest refers to the structured supplementary data associated with the target point of interest, which includes at least the target point of interest's rating, opening hours, and parking information, and may also include information such as the point of interest's address and distance.

[0064] In one alternative approach, the calculated interest score is compared in real time with the first preset vehicle speed and the second preset vehicle speed. Based on the comparison result, the cloud-based interest point database interface is called to retrieve the user rating, business hours, and remaining parking space information of the target interest point. This information is then displayed through a pop-up window on the in-vehicle central control system and simultaneously broadcast via voice, thus completing the lightweight information push.

[0065] Step 142: If the interest score is greater than the second preset score, initiate a destination service recommendation to the passenger.

[0066] Destination service recommendations refer to in-vehicle travel services provided to passengers with a target point of interest as the travel destination, including but not limited to: one-click navigation, route planning, real-time traffic conditions, destination reservations, and other services.

[0067] In one alternative approach, after identifying an interest score greater than a second preset score, the in-vehicle system automatically activates the built-in navigation module to generate multiple planned routes to the target point of interest, proactively pushes one-click navigation and real-time traffic alerts, and completes destination travel service recommendations.

[0068] In this embodiment, the attention intensity levels of passengers are accurately divided by two-level scoring thresholds. For medium-level interest, only simplified basic information is pushed to avoid redundant information from interfering with driving. For high-level interest, in-depth destination travel services are proactively provided to accurately match the potential travel needs of passengers. This achieves graded and differentiated push of services, taking into account both driving safety and service personalization, and effectively improving the adaptability of in-vehicle service recommendations and user experience.

[0069] Furthermore, in one optional approach, the vehicle data also includes vehicle speed; if the vehicle speed is determined to be in a low-speed range, the first preset score and the second preset score remain unchanged; if the vehicle speed is determined to be in a medium-speed range, the first preset score and the second preset score are increased respectively according to a first correction value; if the vehicle speed is determined to be in a high-speed range, the first preset score and the second preset score are increased respectively according to a second correction value. Specifically, the above method also includes: Step 01: If the vehicle speed is less than the first preset speed, maintain the first preset score and the second preset score unchanged.

[0070] Step 02: If the vehicle speed is greater than or equal to the first preset vehicle speed and less than the second preset vehicle speed, increase the first preset score and the second preset score respectively according to the first correction value.

[0071] Step 03: If the vehicle speed is greater than or equal to the second preset vehicle speed, increase the first preset score and the second preset score respectively according to the second correction value.

[0072] The first preset speed refers to a pre-calibrated low-speed threshold, used to classify low-speed driving conditions. This threshold can be set according to urban roads and city traffic scenarios. The second preset speed refers to a pre-calibrated high-speed threshold, a value greater than the first preset speed, used to classify medium-high and high-speed driving conditions. For example, the first preset speed can be set to 30 km / h, and the second preset speed can be set to 80 km / h.

[0073] The first correction value refers to the increment used to increase the first and second preset scores when the vehicle is between a first and a second preset speed. The first correction values ​​for the first and second preset scores can be the same or different; for example, the first correction value can be set to 5 points. The second correction value refers to the increment used to increase the first and second preset scores when the vehicle is above the second preset speed. The second correction values ​​for the first and second preset scores can also be the same or different; for example, the second correction value can be set to 10 points. The second correction value is greater than the first correction value.

[0074] In one optional approach, vehicle speed is collected in real time and compared with a first preset speed. When the vehicle speed is less than the first preset speed, indicating a low-speed range, the locally stored first and second preset scores are retrieved directly without any increment or decrement operations; the original scores are used directly. If the vehicle speed is greater than or equal to the first preset speed but less than the second preset speed, indicating a medium-speed range, a pre-stored first correction value is read, and addition operations are performed on both the first and second preset scores, simultaneously updating the values ​​of both scores to complete the score correction. If the vehicle speed is greater than or equal to the second preset speed, indicating a high-speed range, a second correction value greater than the first correction value is retrieved, and the first and second preset scores are incrementally superimposed to achieve a more significant score increase under high-speed conditions.

[0075] In this embodiment, by dividing the vehicle speed range into low-speed, medium-speed, and high-speed ranges, and adopting the adjustment logic that the higher the vehicle speed, the larger the correction value of the preset score, the more it conforms to the actual driving behavior characteristics that the faster the vehicle travels, the more limited the passenger's field of vision, and the higher the attention to surrounding geographical targets. At low speeds, the baseline score is maintained to avoid excessive push, while at medium and high speeds, the interest score is dynamically improved. This accurately matches the passenger's actual level of attention at different vehicle speeds, and realizes the adaptive optimization of the interest score according to the driving conditions, effectively improving the scenario adaptability and accuracy of subsequent service recommendations.

[0076] Furthermore, in an alternative approach, step 142: if the interest score is greater than the second preset score, initiating a destination service recommendation to the passenger includes: Step 1421: If the interest score is greater than the second preset score, send an inquiry message to the passenger, whereby the inquiry message is used to ask whether to recommend destination services.

[0077] The inquiry information refers to the interactive inquiry prompt information generated by the in-vehicle terminal for the target point of interest, which is used to proactively confirm the passenger's travel intention. It supports two presentation formats: pop-up text prompts and voice broadcast inquiries. The content asks the passenger whether they need to activate the destination service recommendation for the corresponding target point of interest.

[0078] In one optional approach, the in-vehicle terminal compares the passenger's current interest score with a pre-stored second preset score in real time. When the interest score is detected to exceed the second preset score, it is determined that the passenger has a strong interest intention. The terminal then retrieves the identification information of the corresponding target point of interest, generates a standardized voice inquiry statement and a pop-up prompt text, and simultaneously sends the inquiry information to the passenger through a dual interactive approach of voice broadcast through the in-vehicle audio system and screen pop-up display on the in-vehicle terminal.

[0079] Step 1422: Upon receiving positive feedback from the query information, plan the driving route from the vehicle's current location to the target point of interest.

[0080] Positive feedback information refers to valid interactive signals of consent and confirmation from passengers in response to inquiries. This includes feedback data from various in-vehicle interaction methods such as touch screen confirmation, voice command confirmation, and physical button confirmation, and serves as the trigger condition for activating the route planning service.

[0081] The driving route refers to the safe and optimal vehicle route planned and generated by the in-vehicle navigation system by combining multi-dimensional traffic data such as the vehicle's real-time geographical location, the location of the target point of interest, real-time traffic conditions, traffic control, and lane rules. It includes complete navigation data such as route trajectory, turn prompts, mileage, and estimated travel time.

[0082] In one optional approach, the in-vehicle terminal continuously monitors signal data in real time, accurately identifies positive feedback from occupants, and filters out accidental or invalid operation signals. After confirming the receipt of valid positive feedback, it retrieves the vehicle's real-time geolocation coordinates and the coordinates of the target point of interest. Combining this with real-time traffic congestion conditions, road grades, and traffic restrictions, it dynamically calculates and plans the optimal route for the entire journey. Specifically, using the vehicle's real-time geolocation coordinates as the starting point and the target point of interest coordinates as the ending point, it transforms road grades and traffic restrictions into road segment traffic constraints. Simultaneously, it accesses real-time traffic congestion data from the cloud to dynamically allocate traffic weights to each road segment, prioritizing the avoidance of congested road segments and restricted roads, and matching higher-grade roads. After iteratively calculating multiple alternative routes, it selects the route with the shortest travel time and the smoothest route as the final driving path.

[0083] Step 1423: Send the planned driving route to the passengers.

[0084] In one alternative approach, the planned driving route data is visualized and converted into voice information. On the one hand, the route trajectory, road conditions along the way, and travel information are rendered and displayed on the display screen or instrument navigation interface of the vehicle terminal. On the other hand, real-time navigation prompts such as starting, turning, and lane driving are broadcast simultaneously to complete the push display of the driving route.

[0085] In this embodiment, threshold judgment is used to accurately identify the passenger's high-intent target points of interest, abandoning the crude mode of traditional intelligent in-vehicle services that directly and forcibly push navigation. First, by actively inquiring, the subjective travel intentions of passengers are fully respected, effectively avoiding the problems of invalid service pushes interfering with driving and riding and consuming system resources. At the same time, by accurately identifying positive feedback information, it is ensured that the route planning service is only triggered when passengers have a real need, eliminating redundant calculations and erroneous service situations. Finally, by pushing driving routes through multi-terminal linkage, clear and intuitive travel guidance is provided to passengers, realizing humanized and low-interference intelligent destination service push. It not only accurately discovers the potential travel needs of passengers, but also greatly improves the practicality, adaptability and driving experience of in-vehicle intelligent services.

[0086] Reference Figure 6 , Figure 6 A flowchart illustrating another embodiment of the service recommendation method of the present invention is shown, which is executed by a service recommendation device. Figure 6 As shown, in one alternative approach, after step 140: recommending corresponding services to passengers based on interest scores, the method further includes: Step 210: Record passenger feedback behavior data regarding the service, including confirmation behavior, rejection behavior, or ignore behavior; Confirmation behavior refers to the passenger's active acceptance of the service recommendation, including positive actions such as confirming to view details of the target point of interest, agreeing to route planning, and starting navigation. Rejection behavior refers to the passenger's active rejection of the service recommendation, including actively rejecting the recommendation pop-up, rejecting the service via voice, and canceling the navigation recommendation. Ignoring behavior refers to the passive behavior of the passenger not taking any confirmation or rejection action or responding within a preset effective time after the system pushes the service.

[0087] In one alternative approach, the in-vehicle terminal is equipped with a user behavior monitoring and logging module. After each service recommendation is triggered, timed monitoring is started to collect interactive signals such as screen touch, voice commands, and button operations of passengers in real time. According to preset behavior judgment rules, the system distinguishes and marks the confirmation behavior, rejection behavior, and ignore behavior. At the same time, the behavior type, occurrence time, and corresponding recommended service content are stored in a structured manner to form complete passenger feedback behavior data.

[0088] Step 220: If the feedback behavior data is a confirmation behavior, then mark the current multimodal perception data and the interest score of the target interest point as positive samples; In one alternative approach, after identifying a positive interaction behavior for occupant confirmation service, the system automatically retrieves and associates the gaze data, voice data, physiological data, interest score output by the model, and vehicle data such as vehicle speed, vehicle position information, and vehicle posture information corresponding to this inference. These data are then packaged and archived in a unified manner, and positive sample labels are added to store them in the positive samples for incremental training of the model.

[0089] Step 230: If the feedback behavior data is a rejection behavior or an ignore behavior, then mark the current multimodal perception data and the interest score of the target interest point as a negative sample; Positive samples refer to high-quality training samples composed of the current multimodal perception data and interest ratings, labeled with "passenger confirms service," indicating that the model's judgment is consistent with the user's true intention. Negative samples refer to biased training samples composed of the current multimodal perception data and interest ratings, labeled with "passenger refuses or ignores service," indicating that the model's judgment deviates from the user's true intention.

[0090] In one alternative approach, when a passenger is detected to actively refuse service or ignore the action due to timeout, all multimodal perception data and interest scores of the target interest points corresponding to this inference are automatically bound together, uniformly collected and stored, and negative sample labels are added and included in the negative sample dataset for subsequent model error correction.

[0091] Step 240: Fine-tune the multimodal fusion network model using positive and negative samples, and adjust the weights of the corresponding data types in the multimodal perception data according to the triggering scenarios of the negative samples.

[0092] In one alternative approach, positive and negative samples are periodically read in batches and incrementally fine-tuned on the multimodal fusion network model using a small learning rate iterative method to reduce the risk of model overfitting. At the same time, feature tracing is performed on negative sample scenarios to locate the gaze data, voice data, or physiological data types that cause prediction bias. The modal weights of such interference data are adaptively reduced and the weights of effective data are increased to complete the dynamic optimization and update of the model.

[0093] In this embodiment, by collecting real-time service interaction feedback from passengers, the system accurately distinguishes between three types of behaviors: confirmation, rejection, and ignoring, and constructs positive and negative training samples for each. This achieves precise binding between users' subjective intentions and objective perception data, overcoming the drawbacks of traditional fixed model parameters that cannot adapt to users' personalized habits. By continuously fine-tuning the multimodal fusion network through positive and negative samples, and by tracing the source of negative scenarios to correct the weights of each modality's data, the system can continuously correct model prediction biases, weaken invalid interference features, and strengthen effective perception features. This makes interest score inference more closely aligned with the real preferences of different passengers and driving scenarios, enabling online adaptive iterative upgrades of the model. This continuously reduces the probability of service mis-pushing and invalid push notifications, significantly improving the personalization, accuracy, and adaptability of interest point identification and intelligent service recommendation.

[0094] For example, to better illustrate the overall technical solution recommended by the present invention, refer to... Figure 7 , Figure 7 The recommended overall technical solution for serving this invention includes: Acquire multimodal perception data inside the vehicle.

[0095] Based on line-of-sight data, vehicle location information, and vehicle attitude information, the location of the occupant's line of sight in geospatial space is determined; the location of the line of sight is then matched with a pre-stored database of points of interest to obtain the occupant's target points of interest in geospatial space.

[0096] After feature extraction and alignment of the multimodal perception data, the occupant's interest score for the target point of interest is obtained through a multimodal fusion network model. This process includes: extracting features corresponding to gaze data, speech data, and physiological data from the input layer of the multimodal fusion network model; aligning the features corresponding to gaze data, speech data, and physiological data through the feature alignment layer of the multimodal fusion network model; fusing the aligned features through the feature fusion layer of the multimodal fusion network model to obtain fused features, and performing regression prediction on the fused features through a multilayer perceptron (MLP) of the multimodal fusion network model to generate an original score representing the intensity of the occupant's interest; and normalizing the original score through the output layer of the multimodal fusion network model to obtain the occupant's interest score for the target point of interest.

[0097] The vehicle speed is obtained for judgment, and a first preset score value and a second preset score value are determined based on the range in which the vehicle speed falls.

[0098] When the interest score of the target point of interest is greater than or equal to the first preset score and less than the second preset score, the relevant information of the target point of interest is preloaded and cached in the background without being displayed. When the interest score of the target point of interest is greater than or equal to the second preset score, an active service is triggered, generating inquiry information and voice broadcast. After receiving user confirmation, a driving route from the vehicle's current location to the target point of interest is planned; the planned driving route is sent to the occupants; if the user refuses, the refusal event is recorded as a negative sample for model fine-tuning.

[0099] Figure 8 The diagram shows a structural schematic of an embodiment of the service recommendation device of the present invention. The specific embodiments of the present invention do not limit the specific implementation of the service recommendation device.

[0100] like Figure 8 As shown, the service recommendation device may include: processor 402, communications interface 404, memory 406, and communications bus 408.

[0101] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408. Communication interface 404 is used to communicate with other network elements, such as clients or other servers. The processor 402 executes program 410, specifically performing the relevant steps described above in the service recommendation method embodiment.

[0102] Specifically, program 410 may include program code, which includes computer-executable instructions.

[0103] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. The service recommendation apparatus includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0104] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0105] Specifically, program 410 can be invoked by processor 402 to cause the service recommendation device to perform the following operations: Acquire multimodal perception data of the vehicle; Based on multimodal perception data, determine the occupants' points of interest in geospatial targets; Multimodal sensing data is input into a multimodal fusion network model to obtain the occupant's interest rating of the target interest point; Based on interest ratings, corresponding services are recommended to passengers.

[0106] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: Based on line-of-sight data, vehicle location information, and vehicle attitude information, the location of the occupant's line of sight in geographic space is determined. The landing location is matched with a pre-stored database of points of interest to obtain the occupants' target points of interest in the geographic space.

[0107] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: Based on the direction of sight, a sight vector in the occupant coordinate system is constructed, and the sight vector is converted into a first transformation vector in the vehicle coordinate system through a preset rotation matrix from occupant to vehicle. Based on vehicle attitude information, a rotation matrix is ​​constructed from the vehicle coordinate system to the geographic coordinate system, and the first transformation vector is transformed into a second transformation vector in the geographic coordinate system through this rotation matrix. Based on vehicle location information, the second transformation vector, and gaze depth, the location of the occupant's line of sight in geographic space is determined.

[0108] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: Features corresponding to gaze data, speech data, and physiological data are extracted from the input layer of the multimodal fusion network model. The feature alignment layer of the multimodal fusion network model aligns the features corresponding to gaze data, speech data, and physiological data. The aligned features are fused through the feature fusion layer of the multimodal fusion network model to obtain fused features. The fused features are then regressed and predicted by the multilayer perceptron (MLP) of the multimodal fusion network model to generate the original scores representing the intensity of passenger interest. The original scores are normalized by the output layer of the multimodal fusion network model to obtain the passenger's interest rating of the target interest point.

[0109] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: If the interest score is greater than or equal to the first preset score and less than the second preset score, load the relevant information of the target interest point and recommend the relevant information to the passengers. If the interest score is higher than the second preset score, a destination service recommendation will be sent to the passenger.

[0110] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: If the vehicle speed is less than the first preset speed, the first preset score and the second preset score remain unchanged. If the vehicle speed is greater than or equal to the first preset vehicle speed and less than the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the first correction value; If the vehicle speed is greater than or equal to the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the second correction value; The second correction value is greater than the first correction value.

[0111] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: If the interest score is greater than the second preset score, an inquiry message is sent to the passenger, which asks whether to recommend destination services. If a positive feedback message is received regarding the inquiry, a driving route from the vehicle's current location to the target point of interest is planned. Send the planned driving route to the passengers.

[0112] In an alternative approach, program 410 may be invoked by processor 402 to cause the service recommendation device to perform the following operations: Record passenger feedback behavior data regarding the service, including confirmation behavior, rejection behavior, or ignore behavior; If the feedback behavior data is a confirmation behavior, then the multimodal perception data and the interest score of the target interest point in this instance will be marked as positive samples. If the feedback behavior data is a rejection behavior or an ignore behavior, then the multimodal perception data and the interest score of the target interest point in this instance will be marked as negative samples. The multimodal fusion network model is fine-tuned using positive and negative samples, and the weights of corresponding data types in the multimodal perception data are adjusted according to the triggering scenarios of negative samples.

[0113] This invention provides a service recommendation device that addresses the problem in existing technologies where the recommended services deviate from the driver's actual needs due to the inability to accurately capture the driver's implicit needs. Compared to existing technologies, the beneficial effects of the service recommendation device provided by this invention are the same as those of the service recommendation methods provided in the above embodiments, and will not be repeated here.

[0114] This invention provides a computer-readable storage medium storing at least one executable instruction. When the executable instruction is executed on a service recommendation device, it causes the service recommendation device to perform the service recommendation method in any of the above method embodiments.

[0115] Specifically, the executable instructions can be used to cause the service recommendation device to perform the following operations: Acquire multimodal perception data of the vehicle; Based on multimodal perception data, determine the occupants' points of interest in geospatial targets; Multimodal sensing data is input into a multimodal fusion network model to obtain the occupant's interest rating of the target interest point; Based on interest ratings, corresponding services are recommended to passengers.

[0116] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: Based on line-of-sight data, vehicle location information, and vehicle attitude information, the location of the occupant's line of sight in geographic space is determined. The landing location is matched with a pre-stored database of points of interest to obtain the occupants' target points of interest in the geographic space.

[0117] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: Based on the direction of sight, a sight vector in the occupant coordinate system is constructed, and the sight vector is converted into a first transformation vector in the vehicle coordinate system through a preset rotation matrix from occupant to vehicle. Based on vehicle attitude information, a rotation matrix is ​​constructed from the vehicle coordinate system to the geographic coordinate system, and the first transformation vector is transformed into a second transformation vector in the geographic coordinate system through this rotation matrix. Based on vehicle location information, the second transformation vector, and gaze depth, the location of the occupant's line of sight in geographic space is determined.

[0118] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: Features corresponding to gaze data, speech data, and physiological data are extracted from the input layer of the multimodal fusion network model. The feature alignment layer of the multimodal fusion network model aligns the features corresponding to gaze data, speech data, and physiological data. The aligned features are fused through the feature fusion layer of the multimodal fusion network model to obtain fused features. The fused features are then regressed and predicted by the multilayer perceptron (MLP) of the multimodal fusion network model to generate the original scores representing the intensity of passenger interest. The original scores are normalized by the output layer of the multimodal fusion network model to obtain the passenger's interest rating of the target interest point.

[0119] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: If the interest score is greater than or equal to the first preset score and less than the second preset score, load the relevant information of the target interest point and recommend the relevant information to the passengers. If the interest score is higher than the second preset score, a destination service recommendation will be sent to the passenger.

[0120] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: If the vehicle speed is less than the first preset speed, the first preset score and the second preset score remain unchanged. If the vehicle speed is greater than or equal to the first preset vehicle speed and less than the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the first correction value; If the vehicle speed is greater than or equal to the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the second correction value; The second correction value is greater than the first correction value.

[0121] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: If the interest score is greater than the second preset score, an inquiry message is sent to the passenger, which asks whether to recommend destination services. If a positive feedback message is received regarding the inquiry, a driving route from the vehicle's current location to the target point of interest is planned. Send the planned driving route to the passengers.

[0122] In one alternative approach, the executable instructions can be specifically used to cause the service recommendation device to perform the following operations: Record passenger feedback behavior data regarding the service, including confirmation behavior, rejection behavior, or ignore behavior; If the feedback behavior data is a confirmation behavior, then the multimodal perception data and the interest score of the target interest point in this instance will be marked as positive samples. If the feedback behavior data is a rejection behavior or an ignore behavior, then the multimodal perception data and the interest score of the target interest point in this instance will be marked as negative samples. The multimodal fusion network model is fine-tuned using positive and negative samples, and the weights of corresponding data types in the multimodal perception data are adjusted according to the triggering scenarios of negative samples.

[0123] The readable storage medium provided by this invention is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described service recommendation method. This addresses the problem in existing technologies where the recommended services deviate from the driver's actual needs due to the inability to accurately capture the driver's implicit needs. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided by this invention are the same as those of the service recommendation method provided in the above embodiments, and will not be elaborated upon here.

[0124] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other device. Furthermore, the embodiments of this invention are not directed to any particular programming language.

[0125] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of the invention may be practiced without these specific details. Similarly, for the sake of brevity and to aid in understanding one or more aspects of the invention, in the description of exemplary embodiments of the invention above, various features of the embodiments are sometimes grouped together in a single embodiment, figure, or description thereof. The claims, which follow the detailed description, are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of the invention.

[0126] Those skilled in the art will understand that the modules in the device of the embodiment can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiment can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components, except that at least some of such features and / or processes or units are mutually exclusive.

[0127] It should be noted that the above embodiments are illustrative of the invention and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The invention can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names. The steps in the above embodiments, unless otherwise specified, should not be construed as limiting the order of execution.

Claims

1. A service recommendation method, characterized in that, The method includes: Acquire multimodal perception data of the vehicle; Based on the multimodal perception data, the occupant's points of interest in the geospatial area are determined; The multimodal sensing data is input into a multimodal fusion network model to obtain the occupant's interest score for the target point of interest; Based on the interest score, corresponding services are recommended to the passengers.

2. The service recommendation method according to claim 1, characterized in that, The multimodal perception data includes occupant data and vehicle data. The occupant data includes at least occupant gaze data, and the vehicle data includes at least vehicle position information and vehicle attitude information. Determining the occupant's point of interest in the geospatial area based on the multimodal perception data includes: Based on the line-of-sight data, the vehicle location information, and the vehicle posture information, the location of the occupant's line of sight in geographic space is determined. The landing location is matched with a pre-stored database of points of interest to obtain the occupant's target points of interest in the geographic space.

3. The service recommendation method according to claim 2, characterized in that, The gaze data includes gaze direction and gaze depth; determining the location of the occupant's gaze in geographic space based on the gaze data, the vehicle location information, and the vehicle attitude information includes: Based on the line of sight, a line of sight vector in the occupant coordinate system is constructed, and the line of sight vector is converted into a first transformation vector in the vehicle coordinate system through a preset rotation matrix from occupant to vehicle. Based on the vehicle attitude information, a rotation matrix from the vehicle coordinate system to the geographic coordinate system is constructed, and the first transformation vector is converted into a second transformation vector in the geographic coordinate system through the rotation matrix; Based on the vehicle location information, the second transformation vector, and the gaze depth, the location of the occupant's line of sight in the geographic space is determined.

4. The service recommendation method according to claim 1, characterized in that, The multimodal perception data includes occupant data, which includes at least occupant gaze data, speech data, and physiological data; the step of inputting the multimodal perception data into a multimodal fusion network model to obtain the occupant's interest score for the target point of interest includes: The features corresponding to the gaze data, the speech data, and the physiological data are extracted through the input layer of the multimodal fusion network model. The feature alignment layer of the multimodal fusion network model aligns the features corresponding to the gaze data, the speech data, and the physiological data. The aligned features are fused through the feature fusion layer of the multimodal fusion network model to obtain the fused features. The fused features are then regressed and predicted by the multilayer perceptron (MLP) of the multimodal fusion network model to generate the original scores representing the intensity of passenger interest. The original scores are normalized by the output layer of the multimodal fusion network model to obtain the passenger's interest score for the target point of interest.

5. The service recommendation method according to any one of claims 1 to 4, characterized in that, The recommendation of corresponding services to the passenger based on the interest score includes: If the interest score is greater than or equal to the first preset score and less than the second preset score, load the relevant information of the target interest point and recommend the relevant information to the passenger. If the interest score is greater than the second preset score, a destination service recommendation is initiated to the passenger.

6. The service recommendation method according to claim 5, characterized in that, The vehicle data also includes vehicle speed; the method further includes: If the vehicle speed is less than the first preset vehicle speed, the first preset score and the second preset score remain unchanged. If the vehicle speed is greater than or equal to the first preset vehicle speed and less than the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the first correction value; If the vehicle speed is greater than or equal to the second preset vehicle speed, the first preset score and the second preset score are increased respectively according to the second correction value; Wherein, the second correction value is greater than the first correction value.

7. The service recommendation method according to claim 5, characterized in that, If the interest score is greater than the second preset score, the step of initiating a destination service recommendation to the passenger includes: If the interest score is greater than the second preset score, an inquiry message is sent to the passenger, wherein the inquiry message is used to ask whether to recommend destination services; Upon receiving positive feedback from the query information, a driving path is planned from the vehicle's current location to the target point of interest. The planned travel route is sent to the occupants.

8. The service recommendation method according to claim 1, characterized in that, After recommending corresponding services to the passengers based on the interest score, the process further includes: Record the passenger's feedback behavior data regarding the service, wherein the feedback behavior data includes confirmation behavior, rejection behavior, or ignore behavior; If the feedback behavior data is the confirmation behavior, then the current multimodal perception data and the interest score of the target interest point are marked as positive samples; If the feedback behavior data is the rejection behavior or the ignore behavior, then the current multimodal perception data, the interest score of the target interest point, and the vehicle data will be marked as negative samples. The multimodal fusion network model is fine-tuned using the positive and negative samples, and the weights of the corresponding data types in the multimodal perception data are adjusted according to the triggering scenarios of the negative samples.

9. A service recommendation device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the service recommendation method as described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the service recommendation device, causes the service recommendation device to perform the operation of the service recommendation method as described in any one of claims 1-8.

Images