A navigation interaction method, device and computer program product

Abstract

The embodiment of the application discloses a navigation interaction method, comprising: receiving a real scene image and space perception data which are time-synchronized with the real scene image sent by a client, wherein the space perception data comprises at least one of real-time position, motion data and attitude data of the client; determining a current behavior state of a user based on the real scene image and the space perception data; generating a control instruction of a virtual image body based on the current behavior state of the user and a predetermined navigation route; and sending the control instruction to the client, so that the client controls a virtual image body superimposed and displayed in the real scene image to output corresponding interaction feedback based on the control instruction. The scheme not only improves the personalization and interaction intuitiveness of navigation, but also significantly optimizes the use experience of the user in the navigation process through the emotional and visual interaction feedback of the virtual image body.

Description

Technical Field

[0001] This application relates to the field of navigation technology, and in particular to a navigation interaction method, device and computer program product. Background Technology

[0002] With the rapid development of mobile internet and positioning technology, navigation technology has become a core auxiliary tool for users' travel. It has gradually evolved from traditional voice navigation to augmented reality (AR) navigation, providing users with more intuitive and contextualized guidance.

[0003] In existing navigation technologies, traditional navigation primarily outputs fixed prompts such as travel direction and remaining distance via voice or text. This mechanical and rigid guidance format makes it difficult to engage users, leaving them largely in a passive information receiving state. Furthermore, it lacks the ability to dynamically adapt to complex road conditions. For example, in areas with numerous intersections, complex road conditions, or unfamiliar environments, the single information output format can easily lead to a loss of direction, confusion, and reduced decision-making efficiency at key points, resulting in an overall user experience that needs further optimization. Summary of the Invention

[0004] This application provides a navigation interaction method, device, and computer program product.

[0005] This application provides the following solution: According to the first aspect, a navigation interaction method is provided, the method comprising: The system receives real-world scene images sent by a client and spatial perception data synchronized with the real-world scene images in time. The spatial perception data includes at least one of the client's real-time location, motion data, and posture data. Based on the real-world scene image and the spatial perception data, determine the user's current behavioral state; Based on the user's current behavior state and the predetermined navigation route, control instructions for the virtual avatar are generated; The control command is sent to the client, which then controls the virtual avatar superimposed on the real-world scene image to output corresponding interactive feedback based on the control command.

[0006] According to the second aspect, a navigation interaction method is provided, the method being applied to a client, the method comprising: Display real-time acquired images of the real scene, and overlay a virtual avatar onto the real scene images; Acquire spatial perception data of the client that is synchronized with the real-world scene image in time, wherein the spatial perception data includes at least one of the client's real-time location, motion data, and posture data; The system sends the real-world scene image and the spatial perception data to the server and receives control commands from the server for the virtual avatar. The control commands are generated based on the user's current behavior state and a pre-determined navigation route. The user's current behavior state is determined based on the real-world scene image and the spatial perception data. Based on the control commands, the virtual avatar is controlled to output corresponding interactive feedback.

[0007] According to a third aspect, a navigation interaction device is provided, the device being disposed on a server side, the device comprising: The receiving unit is configured to receive a real-world scene image sent by the client and spatial perception data synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time location, motion data, and posture data. The determining unit is configured to determine the user's current behavioral state based on the real-world scene image and the spatial perception data; The generation unit is configured to generate control instructions for the virtual avatar based on the user's current behavior state and a predetermined navigation route; The sending unit is configured to send the control command to the client, so that the client controls the virtual avatar superimposed on the real scene image to output corresponding interactive feedback based on the control command.

[0008] According to a fourth aspect, a navigation interaction device is provided, the device being disposed on a client, the device comprising: The display unit is configured to display real-time acquired images of the real scene and overlay a virtual avatar onto the real scene images; The acquisition unit is configured to acquire spatial perception data of the client that is synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time position, motion data, and posture data. The receiving unit is configured to send the real-world scene image and the spatial perception data to the server, and to receive control instructions from the server regarding the virtual avatar; wherein the control instructions are generated based on the user's current behavior state and a predetermined navigation route, and the user's current behavior state is determined based on the real-world scene image and the spatial perception data. The control unit is configured to control the virtual avatar to output corresponding interactive feedback based on the control commands.

[0009] According to a fifth aspect, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in any one of the first or second aspects above.

[0010] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application embodiment determines the user's current behavioral state by receiving real-world scene images and spatial perception data sent by the client, and generates control instructions for a virtual avatar based on a pre-determined navigation route. The client then controls the virtual avatar superimposed on the real-world scene image to output corresponding interactive feedback, achieving dynamic adaptation between navigation guidance and the user's real-time behavioral state. This not only enhances the personalization and intuitiveness of navigation but also significantly optimizes the user experience during navigation through the interactive feedback of the emotional and visual virtual avatar.

[0011] Of course, any product implementing this application does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a system architecture diagram applicable to the embodiments of this application.

[0014] Figure 2 A flowchart of the navigation interaction method provided in the embodiments of this application.

[0015] Figure 3 A flowchart of the navigation interaction method provided in the embodiments of this application.

[0016] Figure 4 A schematic block diagram of a navigation interaction device provided in an embodiment of this application.

[0017] Figure 5 A schematic block diagram of a navigation interaction device provided in an embodiment of this application.

[0018] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0020] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0022] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0023] In existing navigation technologies, traditional navigation primarily outputs fixed prompts such as travel direction and remaining distance via voice or text. This mechanical and rigid guidance format makes it difficult to engage users, leaving them largely in a passive information receiving state. Furthermore, it lacks the ability to dynamically adapt to complex road conditions. For example, in areas with numerous intersections, complex road conditions, or unfamiliar environments, the single information output format can easily lead to a loss of direction, confusion, and reduced decision-making efficiency at key points, resulting in an overall user experience that needs further optimization.

[0024] In view of this, this application provides a new approach. To facilitate understanding of this application, the system architecture on which this application is based will first be described. Figure 1 An exemplary system architecture that can be applied to embodiments of this application is shown, such as Figure 1 As shown, the system architecture may include a client and a server.

[0025] The server and client are the two main components of an application service. The server, using a server as its primary hardware infrastructure, can include one or more software service modules. In this embodiment, it is mainly responsible for processing the control command logic of the virtual avatar in the navigation interaction service, forming a collaborative front-end and back-end with the client.

[0026] The client's terminal device can include, but is not limited to, smart mobile terminals, wearable devices, and PCs (Personal Computers). Smart mobile devices can include devices such as mobile phones, tablets, laptops, PDAs (Personal Digital Assistants), and connected car terminals. Wearable devices can include devices such as smartwatches, smart glasses, smart bracelets, VR (Virtual Reality) devices, AR (Augmented Reality) devices, and mixed reality devices (devices that support both virtual and augmented reality), etc.

[0027] The client can be a local application located on the terminal device, or it can be a web application opened on the terminal device.

[0028] The server can be a single server, a server cluster consisting of multiple servers, or a cloud server. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Servers (VPS) services, such as high management difficulty and weak service scalability.

[0029] As one feasible embodiment, in a navigation scenario, after the client sends a real-world scene image and spatial perception data to the server, the server determines the control instructions for the virtual avatar using the method provided in this application embodiment, and sends the control instructions to the client, enabling the client to control the virtual avatar to generate corresponding interactive feedback based on the control instructions.

[0030] It should be understood that Figure 1 The number of servers and clients shown is merely illustrative. Depending on implementation needs, there can be any number of servers and clients.

[0031] Figure 2 This is a flowchart of a navigation interaction method provided in an embodiment of this application. The method can be... Figure 1 The server-side execution in the system shown. For example... Figure 2 As shown, the method may include the following steps: Step 201: Receive the real-world scene image and spatial perception data synchronized with the real-world scene image sent by the client. The spatial perception data includes at least one of the following: the client's real-time location, motion data, and posture data.

[0032] Step 202: Determine the user's current behavioral state based on real-world scene images and spatial awareness data.

[0033] Step 203: Based on the user's current behavior state and the predetermined navigation route, generate control instructions for the virtual avatar.

[0034] Step 204: Send control commands to the client, so that the client can control the virtual avatar superimposed on the real scene image to output corresponding interactive feedback based on the control commands.

[0035] As can be seen, this application determines the user's current behavioral state by receiving real-world scene images and spatial perception data sent by the client, and generates control instructions for the virtual avatar in conjunction with a pre-determined navigation route. Based on these control instructions, the client controls the virtual avatar superimposed on the real-world scene image to output corresponding interactive feedback, thereby achieving dynamic adaptation between navigation guidance and the user's real-time behavioral state. This not only enhances the personalization and intuitiveness of navigation, but also significantly optimizes the user experience during navigation through the interactive feedback of the emotional and visual virtual avatar.

[0036] In this embodiment, the virtual avatar refers to a three-dimensional or two-dimensional virtual character model generated based on computer graphics technology and overlaid on an augmented reality navigation scene. This virtual avatar possesses a driveable skeletal animation system and facial expression system, supports speech synthesis output, and can interact with the user in real time through various means such as actions, expressions, and voice. Its appearance style (e.g., cartoon characters, animals, icons), action library, expression library, and voice library can be flexibly configured and updated according to navigation scene requirements or user-personalized settings. As a visual interactive carrier of navigation information, the virtual avatar provides users with intuitive and emotional directional guidance, road condition prompts, and behavioral feedback within real-world scene images. It can also output corresponding facial expression changes, action prompts, and voice guidance based on the user's current behavioral state, thereby achieving personalized interaction and significantly enhancing the fun, intuitiveness, and immersion of the navigation experience.

[0037] The steps in the above process and the effects that can be further produced are described in detail below with reference to the embodiments.

[0038] The following describes in detail step 201, namely, "receiving a real-world scene image sent by the client and spatial perception data synchronized with the real-world scene image in time, wherein the spatial perception data includes at least one of the client's real-time location, motion data, and posture data," with reference to the embodiments.

[0039] In this embodiment, when a user initiates the navigation service through the client, they first input travel information, which includes at least the origin and destination. The server then generates a corresponding navigation route based on this travel information. The user can trigger an augmented reality mode switching command to put the client into augmented reality navigation mode. In this augmented reality navigation mode, the client continuously collects and displays real-time images of the real-world scene and acquires spatial perception data synchronized with the real-world scene images in time.

[0040] The real-world scene images can be acquired in real time through an image acquisition device. This device can be a built-in image acquisition device on the client's terminal device, such as the front or rear camera on the user's smartphone. It can also be an external image acquisition device, such as a high-definition camera connected to the smartphone via USB, Type-C, or Bluetooth. Alternatively, it can be other devices with image acquisition capabilities that communicate with the terminal device, such as the built-in camera of smart glasses or an action camera. The key is to capture the real-world scene around the user in real time and transmit the image data to the client.

[0041] Real-world scene images refer to images of the user's actual surroundings, encompassing various environments the user might be in, such as roads, parks, shopping malls, stores, and scenic spots. In practical applications, as an feasible method, users can activate the image capture function by opening the camera application on their terminal device. At this time, the terminal device's screen will synchronously display the real-world scene captured by the camera in real time, forming a real-time view.

[0042] Spatial-aware data is used to characterize the client's state in physical space and is synchronized with real-world scene images in time. Spatial-aware data may include at least one of the following: Real-time location can be obtained through the Global Navigation Satellite System, or through cellular base station positioning, Wi-Fi positioning, Bluetooth beacon positioning, or a combination of these methods, and is used to determine the client's geographic coordinates, altitude, and positioning accuracy.

[0043] Motion data, which can be acquired through an inertial measurement unit, includes information such as the client's acceleration, angular velocity, composite velocity, and displacement changes in three axes, and is used to determine whether the user is walking, running, stopping, or changing their movement state.

[0044] Attitude data can be obtained through fusion algorithms using attitude sensors (such as gyroscopes, electronic compasses, and accelerometers) or visual inertial odometry. This includes the client's orientation angle, pitch angle, roll angle, and the spatial attitude of the device relative to the ground or a reference coordinate system, which is used to understand the user's line of sight and body posture.

[0045] This spatial awareness data can be collected in real time by sensors built into the terminal device, or obtained through external sensor modules or communication with other positioning devices (such as smartwatches and wearable positioning modules). In practical applications, the client aligns the spatial awareness data with the corresponding real-world scene image and encapsulates the data before sending it to the server.

[0046] In addition, the navigation involved in this application embodiment can be applied to low-speed movement scenarios such as pedestrian navigation, scenic spot guidance, and indoor wayfinding, in which users' needs for perception and interaction with the surrounding environment are more prominent.

[0047] The following describes step 202, namely "determining the user's current behavioral state based on real-world scene images and spatial perception data," in detail with reference to an embodiment.

[0048] After receiving real-world scene images and spatial awareness data, semantic parsing is performed on the real-world scene images to obtain environmental semantic information. This process can be implemented using deep neural network models, such as image understanding models based on convolutional neural networks (CNN), Transformer, or multimodal fusion architectures (e.g., YOLO series object detection models, Mask R-CNN instance segmentation models, etc.). The environmental semantic information characterizes the static and dynamic features of the user's environment and may include at least one of the following: Scene type represents the category of the scene where the user is currently located, such as an intersection, a sidewalk, or a building entrance.

[0049] Obstacle distribution: Identify and locate static obstacles (such as construction fences, parked vehicles, etc.) and dynamic obstacles (such as pedestrians, moving vehicles, etc.) in real-world scene images, and output the spatial coordinates, size, and movement trend of the obstacles (for dynamic obstacles).

[0050] The system detects and extracts landmarks that have navigational reference significance or scene identification significance, such as road signs, shop signs, subway station entrances, and specific public facilities. It also outputs the pixel coordinates and real-world spatial location of the landmarks (based on the mapping relationship between the image coordinate system and the real-world coordinate system).

[0051] After obtaining environmental semantic information, the user's current behavioral state is determined based on the environmental semantic information and spatial perception data.

[0052] The above-mentioned method of extracting environmental semantic information from real-world scene images through semantic parsing can achieve a deep understanding of the user's environment, thereby improving the accuracy of judging the user's current behavioral state and scene adaptability.

[0053] Besides processing real-world scene images through semantic parsing, other methods can be used to process real-world scene images and spatial perception data to determine the user's current behavioral state. For example, feature extraction and fusion can be employed. A pre-trained feature extraction network can extract features from both the real-world scene image and the spatial perception data, obtaining global image feature vectors and spatial perception feature vectors. These two feature vectors can then be fused using an attention mechanism to obtain a fused feature representing the user's environment, which can then be used to determine the user's current behavioral state. Another example is object detection and tracking. A pre-trained object detection model can detect objects such as dynamic obstacles and pedestrians in the real-world scene image. Combined with motion and posture data from the spatial perception data, the detected objects can be tracked, and the user's current behavioral state can be determined based on the object's trajectory and the user's motion state. Yet another example is using an end-to-end deep learning model, directly taking the real-world scene image and spatial perception data as input, and outputting a classification result of the user's current behavioral state. All of these processing methods can achieve an understanding of the user's environment and the determination of their current behavioral state.

[0054] When determining a user's current behavioral state based on environmental semantic information and spatial awareness data, the environmental semantic information and spatial awareness data can be input into a pre-trained behavior prediction model. This model is usually a multimodal classifier based on temporal networks or attention mechanisms. Its output is one or more possible candidate behaviors of the user in the current environment, and a confidence score is assigned to each candidate behavior.

[0055] When the confidence level of a candidate behavior is greater than a preset threshold, the candidate behavior is directly determined as the user's current behavior state. When the confidence level of a candidate behavior is not greater than the preset threshold, it indicates that the model has a large degree of uncertainty in judging behavior in the current environment. At this time, a query instruction corresponding to the candidate behavior will be generated. This query instruction can be presented by the client in the form of voice broadcast or text pop-up. After the client receives the answer information input by the user based on the query instruction, it feeds the answer information back to the server. The server corrects the candidate behavior based on the answer information and finally obtains the current behavior state that is more in line with the user's true intention.

[0056] By inputting environmental semantic information and spatial perception data into a pre-trained behavior prediction model, candidate behaviors and confidence levels are generated. Based on the confidence level threshold judgment rule, the behavior state is quickly determined. At the same time, when the confidence level is insufficient, the candidate behaviors are corrected by generating query instructions and combining user answer information. This balances the efficiency and accuracy of behavior state judgment, avoids the uncertainty of single model prediction, and significantly improves the reliability of the user's current behavior state judgment results and the adaptability of user interaction.

[0057] The following describes in detail step 203, namely "generating control instructions for a virtual avatar based on the user's current behavior state and a predetermined navigation route," with reference to an embodiment.

[0058] In this embodiment of the application, the user's current behavioral state may include, but is not limited to, any one of the following: straight-line state, abnormal direction state, stationary state, and target search state.

[0059] Based on the determined current behavior state and the predetermined navigation route, control instructions for the virtual avatar are generated.

[0060] Specifically, the process of generating control instructions for a virtual avatar may include the following steps: First, based on the predetermined navigation route, determine the current navigation action requirements: Based on the predetermined navigation route, analyze the specific navigation actions that the user needs to perform at the current location. For example, if the user needs to turn right 50 meters ahead on the route, the current navigation action requirement is "prepare to turn right reminder"; if the user has deviated from the planned path, the current navigation action requirement is "route correction guidance".

[0061] Then, based on the user's current behavioral state, the emotional feedback type of the virtual avatar is determined: the user's current behavioral state can map the emotional tone that the virtual avatar should present. For example, when the user is in an abnormal orientation state, the emotional feedback type can be "patient guidance type", and when the user is in a straight-line state, the emotional feedback type can be "relaxed companionship type".

[0062] Finally, based on navigation action requirements and emotional feedback types, control commands for the virtual avatar are generated.

[0063] As one feasible approach, capability data matching the navigation action requirements and emotional feedback types can be selected from the navigation capability data of the virtual avatar, thereby generating specific control commands.

[0064] Navigation capability data is a set of presentation resources that a virtual avatar can access, which includes at least: Action library: Stores a series of action animations that the virtual avatar can execute, such as walking, jumping, pointing, turning, etc. Expression library: Stores a series of facial expressions that the virtual avatar can display, such as smiling, doubt, encouragement, curiosity, etc. Voice library: Stores voice segments or voice synthesis parameters that the virtual avatar can output, covering different navigation scripts.

[0065] Based on navigation action requirements (such as "instructing to turn left") and emotional feedback types (such as "gentle guidance"), the most matching combination can be selected from navigation capability data (e.g., selecting "raise your hand and point to the left" from the action library, "smile" from the expression library, and "please turn left from this side" voice template from the voice library), and packaged to generate structured control commands.

[0066] Optionally, the control commands include at least one of motion control parameters, facial expression control parameters, and voice control parameters.

[0067] Motion control parameters are used to control the type, amplitude, and duration of the virtual avatar's movements. Expression control parameters are used to control the type and intensity of expressions of the virtual avatar; Voice control parameters are used to control the voice and text content of the virtual avatar.

[0068] As an example, if the user is in a state of orientation error and the navigation action is an immediate left turn, the generated control instructions may include: Motion control: Drive the virtual avatar to jump to the left twice in succession and raise its left arm to attract the user's attention and clearly indicate the direction.

[0069] Facial expression control: Make the virtual avatar display a "concerned and determined" expression.

[0070] Voice control: Synthesize and output the following voice: "You need to turn left here, please follow me!"

[0071] Through the above methods, navigation instructions and emotional expression can be deeply integrated, so that the behavior of the virtual avatar can not only accurately convey path information, but also match the user's current emotions and needs, thereby providing a more caring, guiding and immersive navigation interaction experience.

[0072] Furthermore, when selecting capability data from the virtual avatar's navigation capability data that matches navigation action requirements and emotional feedback types, the following steps can also be performed: Obtain the user's cumulative navigation mileage, which is the total mileage accumulated by the user in historical navigation.

[0073] When the cumulative navigation mileage reaches the preset mileage threshold, update the navigation capability data of the virtual avatar, such as adding richer actions, more delicate expressions, or more voices to the virtual avatar.

[0074] In this scenario, when generating control commands, matching capability data is selected from the updated navigation capability data of the virtual avatar based on navigation action requirements and emotional feedback types. This allows users to experience progressively upgraded interactive performance as the frequency of use increases.

[0075] As an example, when a user's cumulative navigation mileage reaches 10 kilometers, the virtual avatar's action library may add an "air flip pointing" action, the expression library may add a "playful wink" expression, and the voice library may add a personalized voice template with the user's nickname. Subsequently, in the same navigation scenario, these newly added capabilities can be invoked to generate more personalized and engaging control commands, such as instructing the virtual avatar to guide direction with an "air flip pointing" action, accompanied by the voice prompt, "XX, turn left ahead and you'll arrive!"

[0076] By introducing a cumulative mileage and capability data update mechanism, the navigation interaction capabilities of the virtual avatar can continuously evolve along with the user's growth, which not only improves the user experience but also further strengthens the emotional connection and companionship experience during the navigation process.

[0077] The following describes in detail step 204, namely "sending control commands to the client so that the client controls the virtual avatar superimposed on the real scene image to output corresponding interactive feedback," with reference to the embodiments.

[0078] After the server generates control instructions for the virtual avatar, it sends them to the client. Upon receiving the control instructions, the client parses and executes them, ultimately driving the virtual avatar superimposed on the real-world scene image to complete the corresponding interactive feedback.

[0079] Optionally, the interactive feedback includes at least one of action feedback, facial expression feedback, and voice feedback corresponding to the control command.

[0080] like Figure 3 As shown in the embodiments of this application, a navigation interaction method is also provided, which can be provided by... Figure 1 The client in the system shown executes this method. The method includes the following steps: Step 301: Display the real-time acquired image of the real scene and overlay the virtual avatar onto the image of the real scene; Step 302: Obtain spatial perception data of the client that is synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time location, motion data, and posture data.

[0081] Step 303: Send the real-world scene image and spatial perception data to the server and receive control commands from the virtual avatar fed back by the server; wherein, the control commands are generated based on the user's current behavior state and a pre-determined navigation route, and the user's current behavior state is determined based on the real-world scene image and the spatial perception data.

[0082] Step 304: Control the virtual avatar to output corresponding interactive feedback based on the control commands.

[0083] In this embodiment, when a user initiates the navigation service through the client, they first input travel information, which includes at least the origin and destination. The server then generates a corresponding navigation route based on this travel information. The user can trigger an augmented reality mode switching command to put the client into augmented reality navigation mode.

[0084] In steps 301 and 302, in augmented reality navigation mode, the client continuously collects and displays real-time acquired images of the real-world scene, and acquires spatial perception data synchronized with the real-world scene images in time. Simultaneously, the client overlays and renders a virtual avatar on the real-world scene images, blending it with the real-world scene to form a real-world scene navigation interface.

[0085] While displaying images of the real-world scene, the client simultaneously collects spatial perception data in real time using built-in or external sensors. This spatial perception data may include at least one of the following: Real-time location can be obtained through the Global Navigation Satellite System, or through cellular base station positioning, Wi-Fi positioning, Bluetooth beacon positioning, or a combination of these methods, and is used to determine the client's geographic coordinates, altitude, and positioning accuracy.

[0086] Motion data, which can be acquired through an inertial measurement unit, includes information such as the client's acceleration, angular velocity, composite velocity, and displacement changes in three axes, and is used to determine whether the user is walking, running, stopping, or changing their movement state.

[0087] Attitude data can be obtained through fusion algorithms using attitude sensors (such as gyroscopes, electronic compasses, and accelerometers) or visual inertial odometry. This includes the client's orientation angle, pitch angle, roll angle, and the spatial attitude of the device relative to the ground or a reference coordinate system, which is used to understand the user's line of sight and body posture.

[0088] In step 303, the real-world scene image and spatial awareness data are encapsulated and sent to the server. Based on the received data, the server executes the user behavior state determination and control instruction generation logic as described in the previous embodiments, generates control instructions for the virtual avatar, and returns them to the client.

[0089] In step 304, after the client receives the control command, the local rendering engine parses and executes the command. Specifically, this includes: Drive the skeletal animation of the virtual avatar according to motion control parameters to perform actions such as walking, jumping, and pointing. The facial expressions of the virtual avatar are adjusted according to the expression control parameters to present emotional states such as smiling, encouragement, and concern; The voice synthesis engine is invoked based on the voice control parameters to generate and play navigation voice with corresponding intonation, speed and emotion; Additionally, it can be used in conjunction with auxiliary visual elements such as voice bubbles, directional arrows, and path highlights.

[0090] All feedback content is spatially overlaid and synchronously presented with real-time real-world scene images through an AR rendering framework, forming a coherent, vivid, and emotional navigation interaction experience.

[0091] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0092] According to another embodiment, a navigation interaction device is also provided, which can be set on the server side. Figure 4 A schematic block diagram of a navigation interaction device according to one embodiment is shown, such as Figure 4 As shown, the device 400 includes: The receiving unit 401 is configured to receive a real-world scene image sent by the client and spatial perception data synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time position, motion data, and posture data. The determining unit 402 is configured to determine the user's current behavioral state based on the real-world scene image and the spatial perception data; The generation unit 403 is configured to generate control instructions for the virtual avatar based on the user's current behavior state and a predetermined navigation route; The sending unit 404 is configured to send the control command to the client, so that the client controls the virtual avatar superimposed on the real scene image to output corresponding interactive feedback based on the control command.

[0093] Optionally, the determining unit 402 is specifically configured as follows: Semantic analysis is performed on the real-world scene image to obtain environmental semantic information, which includes at least one of scene type, obstacle distribution, and landmark location. Based on the environmental semantic information and the spatial perception data, the user's current behavioral state is determined.

[0094] Optionally, the determining unit 402 is specifically configured as follows: The environmental semantic information and the spatial awareness data are input into a pre-trained behavior prediction model to generate the user's candidate behaviors and their confidence levels. When the confidence level of the candidate behavior is greater than a preset threshold, the candidate behavior is determined as the user's current behavior state; When the confidence level of the candidate behavior is not greater than the preset threshold, an inquiry instruction corresponding to the candidate behavior is generated, the answer information input by the user based on the inquiry instruction is received, and the candidate behavior is corrected based on the answer information to obtain the user's current behavior state.

[0095] Optionally, the generation unit 403 is specifically configured as follows: Based on the navigation route, determine the current navigation action requirements; Based on the user's current behavioral state, determine the type of emotional feedback of the virtual avatar; Based on the current navigation action requirements and the emotional feedback type, control instructions for the virtual avatar are generated.

[0096] Optionally, the generation unit 403 is specifically configured as follows: Select capability data that matches the navigation action requirements and the emotional feedback type from the navigation capability data of the virtual avatar; Based on the selected capability data, control instructions for the virtual avatar are generated; The navigation capability data includes at least one of the virtual avatar's action library, expression library, and voice library.

[0097] The device further includes: The mileage acquisition unit 405 is configured to acquire the user's cumulative navigation mileage; when the cumulative navigation mileage reaches a preset mileage threshold, the navigation capability data of the virtual avatar is updated. The generation unit 403 is specifically configured as follows: Based on the navigation action requirements and the emotional feedback type, matching capability data is selected from the updated navigation capability data of the virtual avatar.

[0098] Optionally, the control command includes at least one of motion control parameters, facial expression control parameters, and voice control parameters; The interactive feedback includes at least one of the following: action feedback, facial expression feedback, and voice feedback, which corresponds to the control command.

[0099] Optionally, the user's current behavioral state includes any one of the following: straight-line state, abnormal direction state, stationary state, and target search state.

[0100] According to another embodiment, a navigation interaction device is also provided, which can be set on a client. Figure 5 A schematic block diagram of a navigation interaction device according to one embodiment is shown, such as Figure 5 As shown, the device 500 includes: The display unit 501 is configured to display real-time acquired images of a real scene and to overlay a virtual avatar onto the real scene images; The acquisition unit 502 is configured to acquire spatial perception data of the client that is synchronized with the real scene image in time. The spatial perception data includes at least one of the client's real-time position, motion data, and posture data. The receiving unit 503 is configured to send the real scene image and the spatial perception data to the server, and receive the control instructions of the virtual avatar fed back by the server; wherein the control instructions are generated based on the user's current behavior state and a predetermined navigation route, and the user's current behavior state is determined based on the real scene image and the spatial perception data; The control unit 504 is configured to control the virtual avatar to output corresponding interactive feedback based on the control command.

[0101] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0102] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0103] In addition, embodiments of this application also provide a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0104] And an electronic device, comprising: One or more processors; and A memory associated with the one or more processors, the memory being used to store program instructions that, when read and executed by the one or more processors, perform the steps of the method described in any of the foregoing method embodiments.

[0105] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in any of the foregoing method embodiments.

[0106] in, Figure 6 An exemplary architecture of an electronic device is shown, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0107] The processor 610 can be implemented using a general-purpose CPU, microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits to execute relevant programs and implement the technical solution provided in this application.

[0108] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system 621 for controlling the operation of the electronic device 600, and the basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, a data storage management system 624, a navigation interaction device 400 / 500, etc. The aforementioned navigation interaction device 400 / 500 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when the technical solution provided in this application is implemented through software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0109] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0110] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0111] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0112] It should be noted that although the above-described device only shows the processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, memory 620, bus 630, etc., in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0113] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer program product. This computer program product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0114] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A navigation interaction method, characterized in that, The method includes: The system receives real-world scene images sent by the client and spatial perception data synchronized with the real-world scene images in time. The spatial perception data includes at least one of the client's real-time location, motion data, and posture data. Based on the real-world scene image and the spatial perception data, determine the user's current behavioral state; Based on the user's current behavior state and the predetermined navigation route, control instructions for the virtual avatar are generated; The control command is sent to the client, which then controls the virtual avatar superimposed on the real-world scene image to output corresponding interactive feedback based on the control command.

2. The method according to claim 1, characterized in that, Determining the user's current behavioral state based on the real-world scene image and the spatial perception data includes: Semantic analysis is performed on the real-world scene image to obtain environmental semantic information, which includes at least one of scene type, obstacle distribution, and landmark location. Based on the environmental semantic information and the spatial perception data, the user's current behavioral state is determined.

3. The method according to claim 2, characterized in that, Determining the user's current behavioral state based on the environmental semantic information and the spatial awareness data includes: The environmental semantic information and the spatial awareness data are input into a pre-trained behavior prediction model to generate the user's candidate behaviors and their confidence levels. When the confidence level of the candidate behavior is greater than a preset threshold, the candidate behavior is determined as the user's current behavior state; When the confidence level of the candidate behavior is not greater than the preset threshold, an inquiry instruction corresponding to the candidate behavior is generated, the answer information input by the user based on the inquiry instruction is received, and the candidate behavior is corrected based on the answer information to obtain the user's current behavior state.

4. The method according to claim 1, characterized in that, The control instructions for generating the virtual avatar based on the user's current behavior state and a pre-determined navigation route include: Based on the navigation route, determine the current navigation action requirements; Based on the user's current behavioral state, determine the type of emotional feedback of the virtual avatar; Based on the current navigation action requirements and the emotional feedback type, control instructions for the virtual avatar are generated.

5. The method according to claim 4, characterized in that, The control instructions for generating the virtual avatar based on the navigation action requirements and the emotional feedback type include: Select capability data that matches the navigation action requirements and the emotional feedback type from the navigation capability data of the virtual avatar; Based on the selected capability data, control instructions for the virtual avatar are generated; The navigation capability data includes at least one of the virtual avatar's action library, expression library, and voice library.

6. The method according to claim 5, characterized in that, The method further includes: Obtain the user's cumulative navigation mileage; When the cumulative navigation mileage reaches a preset mileage threshold, the navigation capability data of the virtual avatar is updated; The step of selecting capability data from the navigation capability data of the virtual avatar that matches the navigation action requirements and the emotional feedback type includes: Based on the navigation action requirements and the emotional feedback type, matching capability data is selected from the updated navigation capability data of the virtual avatar.

7. The method according to any one of claims 1 to 6, characterized in that, The control commands include at least one of action control parameters, facial expression control parameters, and voice control parameters; The interactive feedback includes at least one of the following: action feedback, facial expression feedback, and voice feedback, which corresponds to the control command.

8. A navigation interaction method, characterized in that, The method is applied to a client, and the method includes: Display real-time acquired images of the real scene, and overlay a virtual avatar onto the real scene images; Acquire spatial perception data of the client that is synchronized with the real-world scene image in time, wherein the spatial perception data includes at least one of the client's real-time location, motion data, and posture data; The system sends the real-world scene image and the spatial perception data to the server and receives control commands from the server for the virtual avatar. The control commands are generated based on the user's current behavior state and a pre-determined navigation route. The user's current behavior state is determined based on the real-world scene image and the spatial perception data. Based on the control commands, the virtual avatar is controlled to output corresponding interactive feedback.

9. A navigation interactive device, characterized in that, The device is located on the server side, and the device includes: The receiving unit is configured to receive a real-world scene image sent by the client and spatial perception data synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time location, motion data, and posture data. The determining unit is configured to determine the user's current behavioral state based on the real-world scene image and the spatial perception data; The generation unit is configured to generate control instructions for the virtual avatar based on the user's current behavior state and a predetermined navigation route; The sending unit is configured to send the control command to the client, so that the client controls the virtual avatar superimposed on the real scene image to output corresponding interactive feedback based on the control command.

10. A navigation interaction device, characterized in that, The device is located on the client side, and the device includes: The display unit is configured to display real-time acquired images of the real scene and overlay virtual avatars onto the real scene images; The acquisition unit is configured to acquire spatial perception data of the client that is synchronized with the real-world scene image in time. The spatial perception data includes at least one of the client's real-time position, motion data, and posture data. The receiving unit is configured to send the real-world scene image and the spatial perception data to the server, and to receive control instructions from the server regarding the virtual avatar; wherein the control instructions are generated based on the user's current behavior state and a predetermined navigation route, and the user's current behavior state is determined based on the real-world scene image and the spatial perception data. The control unit is configured to control the virtual avatar to output corresponding interactive feedback based on the control commands.

11. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images