Display method, electronic device, and computer program product

Abstract

The embodiment of the application is suitable for the technical field of computers, and provides a display method, an electronic device and a computer program product. The method comprises the following steps: if it is identified that a first target object is contained in video data of an environment where a vehicle currently locates, inputting the video data into a trajectory extraction model corresponding to the first target object, and determining a trajectory feature corresponding to the first target object; inputting the video data into a semantic description model, and determining a text feature corresponding to the first target object; generating prediction information corresponding to the first target object according to the trajectory feature and the text feature; and displaying the prediction information corresponding to the first target object in a head-up display according to first position information of the first target object. Through the method provided in the embodiment, the head-up display can display the prediction information of the first target object, and therefore, the accuracy and availability of virtual information in the head-up display can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a display method, electronic device, and computer program product. Background Technology

[0002] During driving, a driver's decisions are generally based on the status and position of targets such as traffic lights, traffic signs, pedestrians, and other vehicles. Therefore, improving the driver's perception of various targets in the environment is crucial to driving safety. To enhance driver perception, more and more cars are being equipped with Augmented Reality-Head-Up Displays (AR-HUDs). AR-HUDs project virtual information corresponding to targets directly into the driver's field of vision, overlaying the virtual information with the real road conditions. By displaying virtual information corresponding to targets on the AR-HUD, the driver's perception of targets can be improved. For example, by highlighting traffic signs, traffic lights, pedestrians, and other vehicles using AR-HUDs, users can quickly notice the real-time status and position of these targets.

[0003] However, the virtual information displayed in existing augmented reality head-up displays (AR-HUDs) is primarily real-time information collected by sensors, and there may be a time delay between the collection of real-time information and its display on the AR-HUD. Therefore, existing technologies reduce the accuracy of virtual information, making it difficult for users to make decisions about future driving behavior using this delayed virtual information, thus reducing the usability of the virtual information. Summary of the Invention

[0004] In view of this, embodiments of this application provide a display method, electronic device, and computer program product to improve the accuracy and usability of virtual information displayed by a head-up display.

[0005] A first aspect of this application provides a display method, including:

[0006] If a preset first target object is detected in the video data of the current environment of the driving vehicle, the video data is input into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object; the first target object is a movable target object in the video data.

[0007] The video data is input into a semantic description model to determine the text features corresponding to the first target object;

[0008] Based on the trajectory features and the text features, predictive information corresponding to the first target object is generated;

[0009] Based on the first location information of the first target object, the predicted information corresponding to the first target object is displayed on the head-up display.

[0010] In one possible implementation of the first aspect, the prediction information includes the prediction behavior corresponding to the first target object;

[0011] The step of generating prediction information corresponding to the first target object based on the trajectory features and the text features includes:

[0012] The trajectory features and the text features corresponding to the descriptive text are fused to generate fused features;

[0013] The trajectory features, the text features, and the fused features are concatenated to generate concatenated features;

[0014] Based on the splicing features, determine the probability value of the first target object performing each candidate behavior;

[0015] Based on all the probability values, the predicted behavior corresponding to the first target is determined from all the candidate behaviors.

[0016] In one possible implementation of the first aspect, the text feature includes at least one text column vector; the trajectory feature includes at least one trajectory column vector; and the fusion feature includes a first fusion feature and a second fusion feature.

[0017] The step of fusing the trajectory features and the text features to generate fused features includes:

[0018] For any text column vector, calculate the first weighting coefficient between the text column vector and each of the trajectory column vectors;

[0019] The first fusion feature corresponding to the text feature is generated by weighted summation of all the text column vectors according to the first weight coefficient.

[0020] For any trajectory column vector, calculate the second weighting coefficient between the trajectory column vector and each of the text column vectors;

[0021] The second fusion feature corresponding to the trajectory feature is generated by weighting and summing all the trajectory column vectors according to the second weight coefficient.

[0022] In one possible implementation of the first aspect, the formula for calculating the first weighting coefficient between any text column vector and each of the trajectory column vectors is as follows:

[0023]

[0024] in, f represents the first weight coefficient corresponding to the i-th text column vector; text [i] represents the text column vector corresponding to the i-th column; f tranditional [j] represents the trajectory column vector corresponding to the j-th column, where the value of j ranges from [1, k1]; k1 is the total number of trajectory column vectors in the trajectory features; d1 is the preset first attention coefficient; exp represents an exponential function with the natural constant as the base.

[0025] The formula for calculating the second weighting coefficient between any trajectory column vector and each of the text column vectors is as follows:

[0026]

[0027] in, f represents the second weight coefficient corresponding to the trajectory column vector of the j-th column; text [i] represents the text column vector corresponding to the i-th column, where the value of i ranges from [1, k2]; k2 is the total number of text column vectors in the text features; d2 is the preset second attention coefficient.

[0028] In one possible implementation of the first aspect, the video data includes at least one video frame; the first target object includes a pedestrian; and the trajectory features corresponding to the first target object include the walking trajectory features corresponding to the pedestrian.

[0029] The step of inputting the video data into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object includes:

[0030] For any video frame, obtain the skeletal data corresponding to each pedestrian in that video frame;

[0031] For any pedestrian, generate the corresponding skeletal sequence information based on the skeletal data of the pedestrian in all video frames;

[0032] Feature extraction is performed on the skeletal sequence information to generate the walking trajectory features corresponding to any pedestrian.

[0033] In one possible implementation of the first aspect, the skeletal sequence information includes joint sequence information of multiple joint points;

[0034] The step of extracting features from the skeletal sequence information to generate the walking trajectory features corresponding to any pedestrian includes:

[0035] Perform a first convolution operation on the joint sequence information corresponding to each joint point to determine the local motion features corresponding to each joint point.

[0036] The local motion features of all joints are aggregated to determine the cooperative motion features of any pedestrian.

[0037] A second convolution operation is performed on the cooperative motion features to generate the walking trajectory features corresponding to any pedestrian.

[0038] In one possible implementation of the first aspect, the video data includes at least one video frame; the first target includes at least one other vehicle traveling on the road where the driving vehicle is located; the trajectory features corresponding to the first target include the driving trajectory features corresponding to the other vehicle;

[0039] The step of inputting the video data into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object includes:

[0040] For any other vehicle, obtain the second position information of the other vehicle in each video frame of the video data; the second position information is determined by performing target detection on the video data using a target detection algorithm;

[0041] Based on the second position information of the other vehicles in any video frame, the first speed of the other vehicles in any video frame, the second speed of the driving vehicle when capturing the video frame, and the yaw rate of the driving vehicle when capturing the video frame, the trajectory data of the other vehicles in any video frame is determined.

[0042] Feature extraction is performed on the trajectory data of the other vehicles in all the video frames to generate the driving trajectory features corresponding to the other vehicles.

[0043] In one possible implementation of the first aspect, after generating the prediction information corresponding to the first target object based on the trajectory features and the text features, the following steps are included:

[0044] For any first target object, generate a first triplet corresponding to the first target object based on the prediction information corresponding to the first target object and the first location information corresponding to the first target object;

[0045] The status information corresponding to each second target object in the video data is obtained; the second target object is a non-movable target object in the video data; the status information is determined after target detection is performed on the video data by a target detection algorithm;

[0046] For any second target object, generate a second triplet corresponding to the second target object based on the state information corresponding to the second target object and the description text corresponding to the state information;

[0047] Input all the first triples, all the second triples, and the preset knowledge graph into the retrieval model to determine the scene semantic graph corresponding to the video data;

[0048] The scene semantic graph is input into the semantic reasoning model to generate and display the driving strategy information corresponding to the scene semantic graph.

[0049] A second aspect of this application provides a display device, including:

[0050] The trajectory feature acquisition module is used to input the video data into the trajectory extraction model corresponding to the first target object if a preset first target object is detected in the video data of the current environment of the driving vehicle, and to determine the trajectory features corresponding to the first target object; the first target object is a movable target object in the video data.

[0051] The text feature acquisition module is used to input the video data into the semantic description model to determine the text features corresponding to the first target object;

[0052] The prediction module is used to generate prediction information corresponding to the first target object based on the trajectory features and the text features;

[0053] The display module is used to display the prediction information corresponding to the first target object in the head-up display based on the first position information of the first target object.

[0054] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the display method described in the first aspect above.

[0055] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the display method described in the first aspect above.

[0056] A fifth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform the display method described in the first aspect.

[0057] Compared with the prior art, the embodiments of this application have the following advantages:

[0058] In this embodiment, when the electronic device recognizes that the video data of the current environment of the driving vehicle contains a movable first target object, it can generate and display predicted information of the first target object based on the trajectory features and text features corresponding to the first target object. Since the virtual information displayed on the head-up display is the predicted information of the first target object, the method provided in this embodiment can reduce the time delay between the sensor collecting real-time information and displaying the virtual information on the head-up display, thereby improving the accuracy of the virtual information displayed on the head-up display. Furthermore, since the predicted information can represent the possible future situations of the first target object, the driver can obtain the possible future situations of the first target object through the head-up display. Therefore, the method provided in this embodiment can also assist the driver in making decisions about future driving behavior, thereby improving the usability of the virtual information.

[0059] Furthermore, for a movable primary target, trajectory features can intuitively present the target's movement path and speed change trend; while text features can provide supplementary information from the perspectives of category and behavior description, clarifying the current behavioral state of the primary target. Therefore, the predictive information generated by electronic devices by combining these trajectory and text features can improve the accuracy of the electronic devices' predictions of the target's behavior. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0061] Figure 1 This is a schematic diagram of a display method provided in an embodiment of this application;

[0062] Figure 2 This is a schematic diagram of a visual graphic provided in an embodiment of this application;

[0063] Figure 3 This is a schematic diagram of an AR-HUD display screen provided in an embodiment of this application;

[0064] Figure 4This is a schematic diagram of another display method provided in an embodiment of this application;

[0065] Figure 5 This is a schematic diagram of another display method provided in an embodiment of this application;

[0066] Figure 6 This is a schematic diagram of another display method provided in an embodiment of this application;

[0067] Figure 7 This is a schematic diagram of another visual graphic provided in an embodiment of this application;

[0068] Figure 8 This is a schematic diagram of a virtual information display process for an AR-HUD provided in an embodiment of this application;

[0069] Figure 9 This is a schematic diagram of a display device provided in an embodiment of this application;

[0070] Figure 10 This is a schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0071] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0072] The technical solution of this application will be described below through specific embodiments.

[0073] During driving, a driver's decisions are generally based on the state and position of targets such as traffic lights, traffic signs, pedestrians, and other vehicles. Therefore, improving the driver's perception of various targets in the environment is crucial to driving safety. AR-HUD can provide driving cues to the driver by displaying virtual information or highlighting targets in the environment to emphasize them, thus enhancing the driver's environmental perception. Researchers typically categorize the perception capabilities of AR-HUD into three levels: perception level, understanding level, and prediction level.

[0074] The perception level refers to the AR-HUD's ability to acquire and display information collected by sensors. For example, an AR-HUD can directly project the speed of the vehicle the driver is currently driving into the driver's field of vision, allowing the driver to quickly obtain key information without having to look away.

[0075] Understanding the hierarchy refers to AR-HUD's ability to fuse data from multiple sensors, analyze the relationships between various pieces of information, and generate and display the current traffic situation. For example, when it detects a vehicle suddenly braking ahead or a pedestrian crossing the road, AR-HUD displays corresponding warning information in its field of view with prominent colors and icons.

[0076] The prediction level refers to AR-HUD's ability to use historical data and real-time information, along with technologies such as machine learning, to predict future traffic conditions, generate and display predicted information and corresponding driving decision information. For example, in overtaking scenarios, it can predict the acceleration trend of vehicles ahead to provide drivers with the best overtaking opportunity.

[0077] However, the perception capabilities of existing AR-HUDs can only reach the perception and understanding levels, but not the prediction level. That is, existing AR-HUDs cannot predict future situations or provide drivers with driving decision-making information. Therefore, drivers still need to rely on their own experience and subjective judgment to predict the behavior of various objects in the environment, making it difficult for drivers to prepare in advance and greatly reducing driving safety.

[0078] Reference Figure 1 This diagram illustrates a display method provided in an embodiment of this application. The method can be applied to a server. When applied to a server, a wireless communication link can be established between the server and the AR-HUD in the vehicle. The method can also be applied to electronic devices in the vehicle, such as Electronic Control Units (ECUs), Microcontroller Units (MCUs), and Central Processing Units (CPUs). These electronic devices can be external to the AR-HUD or internal to it. Specifically, the display method may include the following steps:

[0079] S101. If the video data of the current environment of the driving vehicle is identified to contain a preset first target object, the video data is input into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object.

[0080] In this embodiment, when the vehicle is started, the electronic device can continuously collect video data of the current environment of the vehicle through the vehicle's onboard camera. The vehicle can be the one housing the head-up display. The onboard camera can be a panoramic camera, a front-view camera, a rear-view camera, etc. After acquiring the video data, the electronic device can identify whether the current environment of the vehicle contains a first target object. If the electronic device identifies the first target object in the video data, it can input the video data into the trajectory extraction model corresponding to the first target object to obtain the trajectory features of the first target object in the video data. Then, it executes the steps S102-S104 of this embodiment to predict the behavior of the first target object and generate prediction information corresponding to the first target object. If the electronic device does not identify the first target object in the video data, it may not perform trajectory prediction for the first target object.

[0081] In one possible implementation, after acquiring video data of the driving vehicle, the electronic device can input the video data into a target detection algorithm to identify all targets in the video data and obtain the detection information corresponding to each target.

[0082] The object detection algorithm can be used to detect a first object and a second object in the video data. The first object can be a movable object in the current environment of the driving vehicle. Specifically, the first object can be a pedestrian, vehicle, or other movable object. The second object can be a non-movable object in the current environment of the driving vehicle. Specifically, the second object can be a traffic sign, traffic light, pedestrian crossing, or other non-movable object. For the first object, the detection information generated by the object detection algorithm can include a detection bounding box, the category label corresponding to the first object, and the first position information of the first object in the video data.

[0083] The first location information may include second location information corresponding to multiple video frames of the first target object. Since video data can contain multiple video frames, and the first target object can appear in some video frames of the video data, a set of locations based on the second location information corresponding to the video frames in which the first target object appears can be obtained, which is the aforementioned first location information.

[0084] For the second target object, the detection information generated by the target detection algorithm may include a detection bounding box, the category label corresponding to the second target object, the state information of the second target object, and the third location information of the second target object in the video data. The category label may include, but is not limited to, vehicle category, pedestrian category, zebra crossing category, traffic light category, traffic sign category, etc.

[0085] For example, when the video data contains vehicles other than the driving vehicle, the detection information for that other vehicle may include a detection box to identify it, the vehicle category, and the second location information of that other vehicle in the video data. When the video data contains a traffic light in a red state, the detection information for that traffic light may include a detection box to identify it, the traffic light category, the red state, and the third location information of that traffic light in the video data.

[0086] It should be noted that since this display method is monitored in real time during the user's driving process, that is, during the acquisition of video data, after the electronic device acquires one or more video frames, it will perform target detection through the above-mentioned target detection algorithm to determine whether it contains the first target object. If the first target object is contained in any video frame, the trajectory of the first target object is predicted through the methods of S101 to S104 in this embodiment to achieve the purpose of predicting the trajectory of the first target object.

[0087] After the electronic device performs target detection using the target detection algorithm, it can determine whether a target object exists in each video frame based on the output of the target detection algorithm. If the target detection algorithm outputs detection information, i.e., a first target object and / or a second target object exist in any video frame, the electronic device can further determine whether the first target object exists in the video data based on the category label in the detection information. If the electronic device determines that the detection information contains the category label corresponding to the first target object, i.e., the electronic device recognizes that the current environment of the driving vehicle contains the first target object, the electronic device can execute the content of S101 to S104 of this embodiment based on the video data to predict the first target object in the video data and display the prediction information corresponding to the first target object. After generating the prediction information corresponding to the first target object, the electronic device can also execute the content of the fourth embodiment of this application based on the prediction information corresponding to the first target object and all detection information to generate and display driving strategy information corresponding to the current environment of the vehicle. If the electronic device determines that none of the detection information contains the category label corresponding to the first target object, that is, the current environment of the driving vehicle only contains the second target object, then the electronic device may not execute the content of S101 to S104 of this embodiment. The electronic device can execute the content of the fourth embodiment of this application according to the detection information corresponding to the second target object to generate and display the driving strategy information corresponding to the current environment of the vehicle.

[0088] If the object detection algorithm does not output detection information, that is, if there is no first object or second object in the video data, the electronic device may not perform behavior prediction for the first object and may not generate driving strategy information. The electronic device may continue to acquire video data of the vehicle's current environment and continue to perform object detection on the acquired video data.

[0089] S102. Input the video data into the semantic description model to determine the text features corresponding to the first target object.

[0090] In this embodiment, for any first target object in the video data, the electronic device can also input the detection information of the first target object and the video data into the semantic description model to generate the text features corresponding to the first target object through the language description model.

[0091] Specifically, the semantic description model can include a text generation algorithm and a text feature extraction algorithm. For example, the text generation algorithm in an electronic device can be a Video-Language Model (Video-LLaMA). The electronic device can first input video data into the text generation algorithm, which can then generate descriptive text. The text generation algorithm can be used to generate descriptive text describing the driving scene in the input video data. The descriptive text generated by the text generation algorithm can describe the information of all targets in the video data and the driving scene information. Then, the electronic device can input the descriptive text corresponding to the video data and the detection information corresponding to the first target object into the text feature extraction algorithm to obtain the text features corresponding to the first target object.

[0092] S103. Based on trajectory features and text features, generate prediction information corresponding to the first target object.

[0093] In this embodiment, for any first target object in the video data, after the electronic device obtains the trajectory features and text features corresponding to the first target object, it can predict the first target object based on the trajectory features and text features and generate prediction information corresponding to the first target object.

[0094] In one possible implementation, after acquiring the trajectory features and text features corresponding to any first target object, the electronic device can first fuse the trajectory features and text features to generate a fused feature corresponding to the first target object. Then, the electronic device can concatenate the trajectory features, text features, and fused feature corresponding to the first target object to generate a concatenated feature corresponding to the first target object. After acquiring the concatenated feature, the electronic device can perform average pooling on the concatenated feature to generate a fixed-length concatenated feature. The electronic device can input the fixed-length concatenated feature into a classifier corresponding to the first target object to determine the probability value of the first target object performing each candidate behavior. The classifier corresponding to the first target object can be trained using a training dataset corresponding to the first target object and candidate behaviors corresponding to the first target object. For example, when the first target object is a vehicle, the classifier corresponding to the first target object can be trained using a training dataset corresponding to vehicles and various candidate behaviors corresponding to vehicles, such as left turn, right turn, deceleration, etc. The electronic device can determine the predicted behavior corresponding to the first target object from all candidate behaviors based on all probability values ​​of the first target object. Specifically, electronic devices can identify the candidate behavior with the highest probability value as the predicted behavior corresponding to the first target object.

[0095] In one possible implementation, the electronic device performs feature fusion of trajectory features and text features as follows: The text features corresponding to the first target object may include at least one text column vector. The trajectory features corresponding to the first target object may include at least one trajectory column vector. The fused features corresponding to the first target object include a first fused feature corresponding to the text features and a second fused feature corresponding to the trajectory features. Specifically, for any text column vector in the text features, the electronic device can calculate a first weighting coefficient between the text column vector and each trajectory column vector. Then, the electronic device can perform a weighted summation of all text column vectors based on the first weighting coefficients corresponding to each text column vector to generate the first fused feature corresponding to the text features. For any trajectory column vector in the trajectory features, the electronic device can calculate a second weighting coefficient between the trajectory column vector and each text column vector. Then, the electronic device can perform a weighted summation of all trajectory column vectors based on the second weighting coefficients corresponding to each trajectory column vector to generate the second fused feature corresponding to the trajectory features.

[0096] In one possible implementation, the formula for calculating the first weight coefficient of any text column vector by the electronic device can be:

[0097]

[0098] in, f can represent the first weight coefficient corresponding to the i-th text column vector; text [i] can represent the text column vector corresponding to the i-th column; f tranditional [j] can represent the trajectory column vector corresponding to the j-th column, and the value range of j can be [1, k1]; k1 can be the total number of trajectory column vectors in the trajectory feature; d1 can be the preset first attention coefficient; exp can represent an exponential function with the natural constant as the base;

[0099] The formula for calculating the second weighting coefficient of any trajectory column vector by an electronic device can be:

[0100]

[0101] in, f can represent the second weight coefficient of the trajectory column vector corresponding to the j-th column; text [i] can represent the text column vector corresponding to the i-th column, and the value of i can be in the range of [1, k2]; k2 can be the total number of text column vectors in the text features; d2 can be the preset second attention coefficient.

[0102] The formula for the weighted summation of all text column vectors by the electronic device can be:

[0103]

[0104] Here, H1 can be the first fusion matrix corresponding to the text features. The first fusion matrix can be expanded as follows:

[0105]

[0106] The formula for calculating the weighted sum of all trajectory column vectors by the electronic device can be:

[0107]

[0108] Here, H2 can be the second fusion matrix corresponding to the trajectory features. The expansion of the second fusion matrix can be expressed as:

[0109]

[0110] In one possible implementation, after determining the first fusion matrix, the second fusion matrix, the text matrix, and the trajectory matrix, the electronic device can input these matrices into a preset splicing function. The splicing function calculates the splicing matrix corresponding to the first target object and performs average pooling on the spliced ​​features to generate average-pooled spliced ​​features. The specific calculation formula for generating the spliced ​​features by the electronic device can be as follows:

[0111] ffused =Paoling(Concat(H1, H2, f)) text f tranditional ))

[0112] Here, `Concat` can represent a concatenation function, and `Pooling` can represent an average pooling function. fused It can represent the splicing features after average pooling.

[0113] In one possible implementation, the specific calculation process by which the electronic device determines the predicted behavior corresponding to the first target object through a classifier can be as follows:

[0114] p = softmax((Wf fused +b)

[0115] class = argmax p

[0116] Here, p can represent a set of probability values, which may contain the probability values ​​of each candidate behavior corresponding to the first target. W can represent the first prediction coefficient. B can represent the second prediction coefficient. Class can represent the predicted behavior corresponding to the first target. Softmax can represent a classifier, which can be used to convert the feature vector into a set of probability values ​​containing the probability values ​​corresponding to each class. argmax can be the maximum value function.

[0117] S104. Display the prediction information corresponding to the first target object in the head-up display based on the first position information of the first target object.

[0118] In this embodiment, after acquiring the prediction information corresponding to any first target object, the electronic device can display the prediction information corresponding to the first target object on the head-up display based on the first position information of the first target object. The prediction information may include the prediction behavior corresponding to the first target object.

[0119] Specifically, when the primary target is a pedestrian, the predicted behavior may include standing, walking, etc. When the primary target is a vehicle other than a driven vehicle, the predicted behavior may include turning left, turning right, slowing down, etc. After determining the predicted behavior corresponding to the primary target, the electronic device can retrieve the corresponding visualization from the database. See also Figure 2 This illustration shows a schematic diagram of a visual graphic provided in an embodiment of this application. For example... Figure 2 As shown in (a), the visualization of pedestrians can include standing figures, walking figures, and figures crossing the road. Figure 2As shown in (b), the vehicle's visualization graphics may include left-turn graphics, right-turn graphics, and deceleration graphics. After acquiring the visualization graphics corresponding to the predicted behavior of the first target object, the electronic device can transmit the first location information and the visualization graphics to the head-up display (HUD). The HUD can project the visualization graphics onto the location of the first target object based on its first location information to display the predicted information corresponding to the first target object. The HUD can be an AR-HUD.

[0120] See Figure 3 This diagram illustrates an AR-HUD display screen provided in an embodiment of this application. Figure 3 As shown, when an electronic device detects that the predicted information of a pedestrian in the video data indicates that the pedestrian is crossing the road, the electronic device can acquire a road-crossing graphic and transmit the road-crossing graphic and the first location information corresponding to the pedestrian to the AR-HUD. The AR-HUD can then overlay the 3D graphic corresponding to the road-crossing graphic onto the pedestrian's location based on the first location information to display the predicted information corresponding to the pedestrian to the driver.

[0121] The method provided in this embodiment allows an electronic device to analyze and predict a first target object in video data and display the predicted information on a head-up display (HUD). For example, when the electronic device predicts that a vehicle ahead may suddenly brake, a pedestrian may run a red light, or a vehicle may emerge from a blind spot, it displays the corresponding predicted information on the HUD in advance. Therefore, the method provided in this embodiment enables drivers to make decisions about future driving behavior based on predicted information, improving the usability of virtual information and reducing the likelihood of accidents. Furthermore, since the method provided in this embodiment can reduce the time delay between the acquisition of real-time information from sensors and the display of the corresponding virtual information on the HUD, it can also improve the accuracy of the virtual information.

[0122] In one possible implementation, the location information corresponding to the first target object can be the first coordinate of the first target object in the world coordinate system. After obtaining the visual graphic corresponding to the first target object and the first coordinate of the first target object, the AR-HUD can convert the first coordinate of the first target object into a second coordinate in the vehicle camera coordinate system, and project the visual graphic corresponding to the first target object onto the two-dimensional screen plane in the driver's view according to the second coordinate.

[0123] Specifically, the calculation process in AR-HUD can be described as follows.

[0124] P AR =T·P c =(x c ,y c ,z c )

[0125]

[0126] P real =||P s -E||

[0127] G AR =f render (P real ,I p )

[0128] F AR =f blend (G AR Scene)

[0129] Among them, P c This can represent the first position information of the first target object in the camera coordinate system. p It can represent the visual graphic corresponding to the first target object. AR =(x c ,y c ,z c (x) can represent the coordinates when an AR-HUD projects 3D graphics. Where x... c This can represent the x-coordinate when an AR-HUD projects a 3D graphic. c It can represent the vertical coordinate when an AR-HUD projects a 3D graphic. c It can represent the vertical coordinates when an AR-HUD projects 3D graphics. T can represent the transformation matrix, used to convert coordinates in the camera coordinate system to coordinates in the vehicle coordinate system. P s This can represent the coordinates of the two-dimensional screen plane from the driver's perspective in the screen coordinate system. x This can represent the focal length of the vehicle-mounted camera along the horizontal axis. y This can represent the focal length of the vehicle-mounted camera along the vertical axis. (c) x ,c y ) can be the coordinates of the optical center of the two-dimensional screen plane. E can represent the projection error coefficient, where the projection error coefficient E can be used to reduce the transformation matrix T and camera parameters, such as f. y and f x The resulting error. G AR It can represent three-dimensional graphics. render This can represent a rendering function, used to render the visualized graphic corresponding to the first target object into a 3D graphic based on the coordinates of the 3D graphic. F AR It can represent the overall image displayed by the AR-HUD. blend This can represent a fusion function used to overlay 3D graphics onto a realistic driving scene. Scene can represent the driving scene displayed by the AR-HUD.

[0130] Figure 4 A flowchart illustrating a specific implementation of a display method S101 provided in a second embodiment of this application is shown. See also... Figure 4 Compared to Figure 1 In the embodiment provided, S101 of the display method includes: S401 to S403, which are detailed below:

[0131] S401. For any video frame, obtain the skeletal data corresponding to each pedestrian in the video frame.

[0132] In this embodiment, the first target object in the video data may include a pedestrian. The trajectory features of the first target object may include the walking trajectory features corresponding to the pedestrian. The trajectory feature extraction model in the electronic device may include a trajectory feature extraction model for extracting the walking trajectory features of pedestrians. When the electronic device recognizes that the detection information output by the target detection algorithm contains detection information with the category label of pedestrian, the electronic device can input the video data into the trajectory extraction model corresponding to the pedestrian category, so as to obtain the walking trajectory features corresponding to each pedestrian in the video data through the trajectory extraction model. Specifically, for any video frame in the video data, the electronic device can obtain the skeletal data corresponding to each pedestrian in the video frame. For example, the electronic device can obtain the skeletal data corresponding to each pedestrian in the video frame through YOLOv8 (You Only Look Once version 8). It should be noted that the electronic device can obtain the skeletal data in the video frame through any deep learning model known to those skilled in the art, and the embodiments of this application are not intended to specifically limit the method of obtaining skeletal data.

[0133] In one possible implementation, the skeletal data may include a set of joints V. t And a skeletal diagram used to represent the connections between various joints. For example, the set of joints may include 17 joints. The skeletal relationship diagram can be represented as an undirected spatial graph. Wherein, the set of joints V... t Relationship diagram with skeleton It can be as follows:

[0134]

[0135]

[0136] in, This can be the skeletal relationship diagram corresponding to the t-th video frame. t It can be the set of key points corresponding to the t-th video frame. It can be the set of skeleton connection edges corresponding to the t-th video frame. This can be the coordinates of the i-th joint in the t-th video frame. N can be the total number of joints; for example, N can be 17.

[0137] S402. For any pedestrian, generate the corresponding skeletal sequence information based on the skeletal data of any pedestrian in all video frames.

[0138] In this embodiment, for any pedestrian, the electronic device can sequentially connect the skeletal data of the pedestrian in all video frames according to the order of the video frames to generate the skeletal sequence information corresponding to the pedestrian. Specifically, the electronic device can first sequentially connect the sets of joints corresponding to each video frame according to the order of the video frames to generate joint sequence information. The electronic device can also sequentially connect the sets of skeletal connection edges corresponding to each video frame according to the order of the video frames to generate connection edge sequence information. Then, the electronic device can generate skeletal sequence information based on the joint sequence information and the connection edge sequence information.

[0139] The specific calculation process by which an electronic device generates skeletal sequence information can be described as follows:

[0140] V = U t V t

[0141]

[0142] G = (V, E)

[0143] Here, V can represent the joint sequence information formed by sequentially connecting multiple sets of joint points. E can represent the connection edge sequence information formed by sequentially connecting multiple sets of bone connection edges. t This can be a temporal connection edge used to connect the same joint points from frame t-1 to frame t. G can represent skeletal sequence information.

[0144] S403. Extract features from the skeletal sequence information to generate the walking trajectory features for any pedestrian.

[0145] In this embodiment, after the electronic device obtains the skeletal sequence information corresponding to any pedestrian in the video data, it can extract features from the skeletal sequence information of the pedestrian to generate the walking trajectory features corresponding to the pedestrian.

[0146] Specifically, the skeletal sequence information can include joint sequence information of multiple joints. After acquiring the skeletal sequence information corresponding to any pedestrian, the electronic device can first perform a first convolution operation on the joint sequence information corresponding to each joint in the skeletal sequence information to determine the local motion features corresponding to each joint. Then, the electronic device performs feature aggregation on the local motion features of all joints in the skeletal sequence information to determine the pedestrian's coordinated motion features. Among them, the coordinated motion features can be used to represent the pattern of coordinated movement of multiple different joints. Finally, the electronic device can perform a second convolution operation on the pedestrian's coordinated motion features to generate the pedestrian's corresponding walking trajectory features.

[0147] In one possible implementation, the algorithm for feature extraction from skeletal sequence information in an electronic device can be an algorithm composed of three different levels of ST-GCN networks. This algorithm can be trained using the Maximum Mean Discrepancy (MMD) function. Specifically, the first-level ST-GCN network can capture the motion patterns corresponding to each joint in the skeletal sequence information to generate local motion features for each joint. The second-level ST-GCN network can extract higher-level features from the local motion features to capture complex motion patterns of multiple joints coordinating, generating cooperative motion features. Finally, the third-level ST-GCN network can capture the global motion intent based on the cooperative motion features, outputting a high-dimensional feature, namely the pedestrian's walking trajectory feature.

[0148] Specifically, the calculation process of the three-level ST-GCN network can be described as follows:

[0149]

[0150]

[0151]

[0152] Among them, h( 0 ) can represent the input features of the first-level ST-GCN network, namely, the skeletal sequence information. h (1) This can represent the output features of the first-level ST-GCN network, i.e., local motion features. For example, That is, h (1) It can be an N×d l The matrix. Where, It can represent the set of real numbers; d l It can be the feature dimension corresponding to the first-level ST-GCN network; N can be the number of nodes in the skeletal sequence information. Ak This can represent the predefined k-th adjacency matrix, where... This can be represented as the k-th weight matrix in the first-level ST-GCN network, where... σ can represent an activation function; an example of σ is a rectified linear unit (ReLU). (2) This can represent the output characteristics of the second-level ST-GCN network, namely, the cooperative motion characteristics. This can represent the k-th weight matrix in the second-level ST-GCN network. (3) This can represent the output characteristics of the third-level ST-GCN network, namely the walking trajectory characteristics. It can represent the k-th weight matrix in the third-level ST-GCN network.

[0153] The method provided in this embodiment allows electronic devices to acquire pedestrian trajectory features based on the pedestrian's skeletal data. Since skeletal data directly reflects the position and movement state of the pedestrian's joints, and the movement trajectories of these joints constitute the core of the pedestrian's movement trajectory, acquiring pedestrian trajectory features through skeletal data more accurately captures the pedestrian's movement trends and details compared to methods based solely on image pixels or target contours. Therefore, the method provided in this embodiment can improve the accuracy of trajectory features.

[0154] Furthermore, in this embodiment, the electronic device can extract features from skeletal data using a three-layer ST-GCN network. Each layer of the ST-GCN network can model the spatial information in the skeletal data, automatically learning the spatial relationships between skeletal joints, such as the relative positions and connection methods between joints in different parts of the human body, thereby extracting more representative spatial features. Therefore, by stacking three layers of ST-GCN networks, the spatial features in the skeletal data can be mined more deeply and comprehensively, capturing the spatial and temporal information of human posture more effectively than traditional methods, thus improving the accuracy of trajectory features.

[0155] Figure 5 A flowchart illustrating a specific implementation of a display method S101 provided in the third embodiment of this application is shown. See also... Figure 5 Compared to Figure 1 In the embodiment provided, S101 of the display method includes: S501 to S503, which are detailed below:

[0156] S501. For any other vehicle, obtain the second position information of the other vehicle in each video frame of the video data.

[0157] In this embodiment, the first target object in the video data may include vehicles other than the driving vehicle. The trajectory features of the first target object may include the driving trajectory features corresponding to other vehicles. The trajectory feature extraction model in the electronic device may include a trajectory feature extraction model for extracting the driving trajectory features of other vehicles. When the electronic device recognizes that the detection information output by the target detection algorithm contains detection information with a category label of vehicle category, the electronic device can input the video data into the trajectory extraction model corresponding to the vehicle category, so as to obtain the driving trajectory features corresponding to each other vehicle in the video data through the trajectory extraction model.

[0158] Specifically, for any vehicle other than the driving vehicle in the video data, the acquiring electronic device can obtain the second position information of the vehicle in each video frame of the video data through the target detection algorithm.

[0159] S502. Based on the second position information of other vehicles in any video frame, the first speed of other vehicles in any video frame, the second speed of the driving vehicle when capturing any video frame, and the yaw rate of the driving vehicle when capturing any video frame, determine the trajectory data of other vehicles in any video frame.

[0160] In this embodiment, after the electronic device obtains the second position information of the vehicle in each video frame of the video data, for any other vehicle in any video frame, the electronic device can determine the trajectory data of the vehicle in the video frame based on the second position information of the vehicle in the video frame, the first speed of the vehicle in the video frame, the second speed of the driving vehicle when the video frame was captured, and the yaw rate of the driving vehicle when the video frame was captured.

[0161] Specifically, the trajectory data of a vehicle in a certain video frame can be expressed as follows:

[0162]

[0163] Among them, X t (n) This can represent the trajectory data of the nth vehicle in the tth video frame. It can represent the second position information of the nth vehicle in the tth video frame, that is, the coordinates of the nth vehicle in the tth video frame. This can represent the first speed of the nth vehicle in the tth video frame. t This can represent the second speed of the vehicle when the video frame was captured. It can represent the yaw rate of the vehicle when the video frame was captured.

[0164] S503. Extract features from the trajectory data of other vehicles in all video frames to generate driving trajectory features corresponding to other vehicles.

[0165] In this embodiment, after acquiring the trajectory data of all video frames of any other vehicle in the video data, the electronic device extracts features from all trajectory data corresponding to the other vehicle to generate driving trajectory features corresponding to that other vehicle. For example, the algorithm used by the electronic device to extract features from the vehicle's trajectory data can be a Long Short-Term Memory (LSTM) network.

[0166] In one possible implementation, the electronic device can first use an encoder to encode the trajectory data corresponding to each video frame sequentially according to the order of the video frames, in order to obtain the time-series characteristics of the vehicle in the video data. Specifically, the encoder can use a recursive function to encode the trajectory data corresponding to each video frame sequentially.

[0167] The encoder's calculation formula can be as follows:

[0168] h t ,c t =LSTM(X) t-1 (n) h t-1 c t-1 )

[0169]

[0170] Among them, h t This can represent the hidden state of the t-th video frame. Where d is the vector dimension of the hidden state. The hidden state can be used to pass the time series features corresponding to the current t-th video frame to the time series features corresponding to the next video frame. t X is a memory unit that can represent the t-th video frame, storing time-series features for a long time and helping the encoder handle the dependencies between video frames in a long sequence. t-1 (n) This can represent the trajectory data of the nth vehicle in the (t-1)th video frame. N Z can represent the time-series features output by the encoder after encoding the last video frame. (n) This can represent the time series features corresponding to the nth vehicle.

[0171] Then, the electronic device can decode the time-series features through a decoder to obtain the driving trajectory features of the vehicle in the video data.

[0172] The decoder expression can be as follows:

[0173]

[0174] in, It can represent the driving trajectory features corresponding to the nth vehicle. LSTM-Decoder can represent the encoder in LSTM.

[0175] Specifically, the expression for the driving trajectory characteristics of the nth vehicle can be as follows:

[0176]

[0177] in, It can represent the driving trajectory features of the nth vehicle in the first video frame.

[0178] The method provided in this embodiment allows the electronic device to comprehensively consider the position and speed of other vehicles, as well as its own vehicle's speed and yaw rate, when acquiring the driving trajectory characteristics of other vehicles. Furthermore, since the position and speed of other vehicles can be used to determine the vehicle's spatial coordinates and motion trends to help the electronic device calibrate its trajectory, and the driving vehicle's own speed and yaw rate can assist the electronic device in trajectory analysis from a relative motion perspective, the method provided in this embodiment can improve the accuracy of driving trajectory characteristics.

[0179] Figure 6 A flowchart illustrating a specific implementation of a display method S103 provided in the fourth embodiment of this application is shown. See also... Figure 6 Compared to Figure 1 In the embodiment provided, the display method after S103 includes: S601 to S603, which are detailed below:

[0180] S601. If there is a first target and a second target in the video data, then the driving strategy information corresponding to the scene semantic graph is generated and displayed based on the first triplet corresponding to the first target, the second triplet corresponding to the second target, and the preset knowledge graph.

[0181] In this embodiment, after the electronic device performs target detection on the video data using a target detection algorithm, it can determine whether a first target object and / or a second target object exist in the video data based on the output of the target detection algorithm. If neither the first target object nor the second target object exists in the video data, the electronic device may not generate driving strategy information. If the electronic device determines that the first target object and the second target object exist in the video data, it can obtain the first triplet corresponding to the first target object and the second triplet corresponding to the second target object. Then, the electronic device can input all the first triplets, all the second triplets, and the knowledge graph pre-set by the R&D personnel into the retrieval model to determine the scene semantic graph corresponding to the video data. After obtaining the scene semantic graph through the retrieval model, in order to improve the uniformity of the scene semantic graphs generated by the electronic device based on different video data, the electronic device can also input the scene semantic graph into a large language model to aggregate the scene semantic graph through the large language model, forming a unified scene semantic graph.

[0182] It should be noted that the retrieval model in the electronic device can be any model known to those skilled in the art capable of retrieving the semantic graph corresponding to triples from a knowledge graph, such as a graph embedding-based retrieval model or a graph neural network-based retrieval model. This application embodiment is not intended to specifically limit the retrieval model. The scene semantic graph can include a neighborhood semantic graph and a path semantic graph. The path semantic graph is a semantic graph constructed by the electronic device based on a specific path. The path in the path semantic graph is a sequence of nodes determined by the electronic device in the knowledge graph according to all first triples, all second triples, and preset rules. The path semantic graph is a semantic graph constructed by extracting all adjacent nodes of each central node and the edges between these nodes, with each triple as the central node.

[0183] Specifically, the expressions for the neighborhood semantic graph, path semantic graph, and scene semantic graph can be shown below:

[0184] V q ={v1,v2,...,v n}

[0185] G path ={(v i ,r k ,v j )v j ,v j ∈V q ,r k ∈R, path length ≤ k}

[0186] G neighbor ={(v i ,r′ k,v′ j )v′ j ,v′ j ∈ Domain (v i ),r k ∈R}

[0187] G p ={G path G neighbor}

[0188] G m =LLM(G p )

[0189] Among them, V q G can represent the set of entities consisting of all query entities in all first triples and all second triples. path It can represent a path semantic graph. i This can represent the i-th query entity in the entity set. j r can represent the j-th query entity in the entity set. k It can represent the query entity v i and query entity v j The connections between these entities. R can represent the set of all connections. k can be a preset path length threshold used to limit the path length from the query entity v. i To query entity v j The path length. G neighbor It can represent a neighborhood semantic graph. j ' can represent the query entity v i Queries within the neighborhood of r k ' can represent the query entity v i and query entity v j The relationship between ′. G p It can represent a scene semantic graph composed of a neighborhood semantic graph and a path semantic graph. LLMs can represent large language models. m It can represent a scene semantic graph after aggregation processing through a large language model.

[0190] After acquiring the scene semantic map corresponding to the video data, the electronic device can input the scene semantic map into a semantic reasoning model to generate driving strategy information corresponding to the scene semantic map. Then, the device can retrieve the visualization graphic corresponding to the driving strategy information from the database and transmit the visualization graphic and the third position information corresponding to the driving strategy information to a head-up display (HUD) for displaying the driving strategy information. The third position information corresponding to the driving strategy information can be pre-defined by the developers. Specifically, for the method of displaying driving strategy information on the HUD, the reader can refer to the content of the first embodiment of this application, replacing "predicted information corresponding to the first target object" with "driving strategy information" and "first position information of the first target object" with "third position information corresponding to the driving strategy information" to understand the specific method of displaying driving strategy information on the HUD in this embodiment. See also... Figure 7 This illustrates a schematic diagram of another visual graphic provided in an embodiment of this application. For example... Figure 7 The visualizations corresponding to the driving strategy information shown may include steering graphs, slow-moving graphs, and deceleration graphs. For example... Figure 3 As shown, when the driving strategy information generated by the electronic device is deceleration, the electronic device can acquire a deceleration graphic and transmit the deceleration graphic and the corresponding third-position information of the driving strategy information to the AR-HUD. The AR-HUD can then overlay the three-dimensional graphic corresponding to the deceleration graphic onto the corresponding position based on the third-position information to display the driving strategy to the driver.

[0191] In one possible implementation, the electronic device can first execute the content of the first embodiment of this application based on the video data to obtain prediction information corresponding to each first target object. Then, the electronic device can generate a first triplet corresponding to the first target object based on the prediction information corresponding to the first target object. Specifically, for any first target object, the electronic device can generate a first triplet corresponding to the first target object based on the prediction information corresponding to the first target object and the first position information in the detection information corresponding to the first target object.

[0192] For any second target object, after acquiring the detection information corresponding to the second target object, the electronic device can retrieve the description text corresponding to the state information from the database based on the state information in the detection information. Then, the electronic device can generate a second triplet corresponding to the second target object based on the state information and the description text corresponding to the state information.

[0193] S602. If the video data contains only the first target object, then the driving strategy information corresponding to the scene semantic graph is generated and displayed based on the first triplet corresponding to the first target object and the preset knowledge graph.

[0194] In this embodiment, if the electronic device determines that a first target object exists in the video data, the electronic device can generate and display driving strategy information corresponding to the scene semantic graph based on the first triplet corresponding to the first target object and a preset knowledge graph. The specific method for the electronic device to generate the first triplet and driving strategy information is described in S601 of this embodiment, and will not be repeated here.

[0195] S603. If the video data contains only a second target object, then the driving strategy information corresponding to the scene semantic graph is generated and displayed based on the second triplet corresponding to the second target object and the preset knowledge graph.

[0196] In this embodiment, if the electronic device determines that a second target object exists in the video data, the electronic device can generate and display driving strategy information corresponding to the scene semantic graph based on the second triplet corresponding to the second target object and a preset knowledge graph. The specific method for the electronic device to generate the second triplet and driving strategy information is described in S601 of this embodiment, and will not be repeated here.

[0197] The method provided in this embodiment allows the electronic device to generate driving strategy information based on the presence of targets in the video data, thereby improving the usability of the driving strategy information. Furthermore, when both a first target and a second target exist simultaneously in the video data, the electronic device can simultaneously combine the triples of the first and second targets to construct a scene semantic graph. Therefore, the method provided in this embodiment enables the electronic device to comprehensively perceive potential risks in the driving environment, thereby improving the adaptability of the driving strategy to the scene, providing the driver with more realistic decision support, and enhancing the usability of the driving strategy.

[0198] See Figure 8 This illustration shows a schematic diagram of a virtual information display process for an AR-HUD provided in an embodiment of this application. For example... Figure 8As shown, after acquiring video data, the electronic device can first perform scene perception based on the video data to understand the current driving scenario of the vehicle. The electronic device can then predict the intent of each primary target in the driving scenario. Specifically, the electronic device can use an ST-GCN network and a semantic description model to obtain pedestrian trajectory features corresponding to each pedestrian in the video data, and analyze and predict the behavior of each pedestrian based on these features. The electronic device can also use an LSTM model and a semantic description model to obtain driving trajectory features corresponding to each vehicle in the video data, and analyze and predict the behavior of each vehicle based on these features. Finally, the electronic device can use object detection algorithms to obtain secondary targets in the driving scenario, as well as the state information corresponding to each secondary target.

[0199] Then, the electronic device can use a pre-set knowledge graph and large language model to perform collaborative reasoning on the predicted information of the first target object and the state information of the second target object to generate driving strategy information. The electronic device can acquire visual representations corresponding to the predicted behavior of the vehicle, the predicted behavior of the pedestrian, and the driving strategy information, and send the visual representations to the AR-HUD. The AR-HUD can project the visual representations onto the screen in the driver's view according to a pre-set projection algorithm to provide the driver with predicted information and driving strategy information.

[0200] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0201] Reference Figure 9 The diagram illustrates a 903 device according to an embodiment of this application, which may specifically include a trajectory feature acquisition module 901, a text feature acquisition module 902, a prediction module 903, and a display module 904, wherein:

[0202] The trajectory feature acquisition module 901 is used to input the video data into the trajectory extraction model corresponding to the first target object if a preset first target object is detected in the video data of the current environment of the driving vehicle, and to determine the trajectory features corresponding to the first target object; the first target object is a movable target object in the video data.

[0203] The text feature acquisition module 902 is used to input the video data into the semantic description model to determine the text features corresponding to the first target object;

[0204] The prediction module 903 is used to generate prediction information corresponding to the first target object based on the trajectory features and the text features;

[0205] Display module 904 is used to display the prediction information corresponding to the first target object in the head-up display according to the first position information of the first target object.

[0206] The prediction module can also be used to perform feature fusion on the trajectory features and the text features corresponding to the description text to generate fused features; to perform feature concatenation on the trajectory features, the text features and the fused features to generate concatenated features; to determine the probability value of the first target object performing each candidate behavior based on the concatenated features; and to determine the predicted behavior corresponding to the first target object from all the candidate behaviors based on all the probability values.

[0207] The prediction module can also be used to calculate, for any text column vector, a first weight coefficient between the text column vector and each of the trajectory column vectors; perform a weighted summation of all the text column vectors based on the first weight coefficient to generate the first fusion feature corresponding to the text feature; and calculate, for any trajectory column vector, a second weight coefficient between the trajectory column vector and each of the text column vectors; perform a weighted summation of all the trajectory column vectors based on the second weight coefficient to generate the second fusion feature corresponding to the trajectory feature.

[0208] The calculation formula in the prediction module can be:

[0209]

[0210] in, f represents the first weight coefficient corresponding to the i-th text column vector; text [i] represents the text column vector corresponding to the i-th column; f tranditional [j] represents the trajectory column vector corresponding to the j-th column, where the value of j ranges from [1, k1]; k1 is the total number of trajectory column vectors in the trajectory features; d1 is the preset first attention coefficient; exp represents an exponential function with the natural constant as the base.

[0211] The formula for calculating the second weighting coefficient between any trajectory column vector and each of the text column vectors is as follows:

[0212]

[0213] in, f represents the second weight coefficient of the trajectory column vector corresponding to the j-th column; text[i] represents the text column vector corresponding to the i-th column, where the value of i ranges from [1, k2]; k2 is the total number of text column vectors in the text features; d2 is the preset second attention coefficient.

[0214] The trajectory feature acquisition module can also be used to acquire the skeletal data corresponding to each pedestrian in any video frame; for any pedestrian, generate the skeletal sequence information corresponding to the pedestrian based on the skeletal data of the pedestrian in all video frames; and extract features from the skeletal sequence information to generate the walking trajectory feature corresponding to the pedestrian.

[0215] The trajectory feature acquisition module can also be used to perform a first convolution operation on the joint sequence information corresponding to each joint point to determine the local motion features corresponding to each joint point; perform feature aggregation on the local motion features of all joint points to determine the cooperative motion features of any pedestrian; and perform a second convolution operation on the cooperative motion features to generate the walking trajectory features corresponding to any pedestrian.

[0216] The trajectory feature acquisition module can also be used to acquire, for any other vehicle, the second position information of the other vehicle in each video frame of the video data; the second position information is determined after target detection of the video data by a first target detection algorithm; based on the second position information of the other vehicle in any video frame, the first speed of the other vehicle in any video frame, the second speed of the driving vehicle when shooting the video frame, and the yaw rate of the driving vehicle when shooting the video frame, the trajectory data of the other vehicle in any video frame is determined; features are extracted from the trajectory data of the other vehicle in all the video frames to generate the driving trajectory features corresponding to the other vehicle.

[0217] The display module can also be used to generate a first triplet corresponding to any first target object based on the prediction information and the first position information of the first target object; obtain the state information of each second target object in the video data; the second target object is an immovable target object in the video data; the state information is determined after target detection of the video data by a target detection algorithm; for any second target object, generate a second triplet corresponding to the second target object based on the state information and the description text corresponding to the state information; input all the first triplets, all the second triplets, and a preset knowledge graph into the retrieval model to determine the scene semantic graph corresponding to the video data; input the scene semantic graph into the semantic reasoning model to generate and display the driving strategy information corresponding to the scene semantic graph.

[0218] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0219] Reference Figure 10 The diagram illustrates an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device 1000 in this embodiment includes: a processor 1010, a memory 1020, and a computer program 1021 stored in the memory 1020 and executable on the processor 1010. When the processor 1010 executes the computer program 1021, it implements the steps in the various embodiments of the above-described display method, for example... Figure 1 The steps S101 to S105 are shown. Alternatively, when the processor 1010 executes the computer program 1021, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 9 The functions of modules 901 to 905 are shown.

[0220] For example, the computer program 1021 can be divided into one or more modules / units, which are stored in the memory 1020 and executed by the processor 1010 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which can be used to describe the execution process of the computer program 1021 in the electronic device 1000. For example, the computer program 1021 can be divided into a trajectory feature acquisition module, a text feature acquisition module, a prediction module, and a display module, with the specific functions of each module as follows:

[0221] The trajectory feature acquisition module 901 is used to input the video data into the trajectory extraction model corresponding to the first target object if a preset first target object is detected in the video data of the current environment of the driving vehicle, and to determine the trajectory features corresponding to the first target object; the first target object is a movable target object in the video data.

[0222] The text feature acquisition module 902 is used to input the video data into the semantic description model to determine the text features corresponding to the first target object;

[0223] The prediction module 903 is used to generate prediction information corresponding to the first target object based on the trajectory features and the text features;

[0224] Display module 904 is used to display the prediction information corresponding to the first target object in the head-up display according to the first position information of the first target object.

[0225] The electronic device 1000 can be a desktop computer, cloud server, or other computing device. The electronic device 1000 may include, but is not limited to, a processor 1010 and a memory 1020. Those skilled in the art will understand that... Figure 10 This is merely one example of electronic device 1000 and does not constitute a limitation on electronic device 1000. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device 1000 may also include input / output devices, network access devices, buses, etc.

[0226] The processor 1010 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0227] The memory 1020 can be an internal storage unit of the electronic device 1000, such as a hard disk or memory of the electronic device 1000. The memory 1020 can also be an external storage device of the electronic device 1000, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1000. Furthermore, the memory 1020 can include both internal and external storage units of the electronic device 1000. The memory 1020 is used to store the computer program 1021 and other programs and data required by the electronic device 1000. The memory 1020 can also be used to temporarily store data that has been output or will be output.

[0228] This application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the display method as described in the foregoing embodiments.

[0229] This application also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the display methods described in the foregoing embodiments.

[0230] This application also discloses a computer program product that, when run on a computer, causes the computer to execute the display methods described in the foregoing embodiments.

[0231] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A display method characterized by comprising: include: If a preset first target object is detected in the video data of the current environment of the driving vehicle, the video data is input into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object; the first target object is a movable target object in the video data. The video data is input into a semantic description model to determine the text features corresponding to the first target object; Based on the trajectory features and the text features, predictive information corresponding to the first target object is generated; Based on the first location information of the first target object, the predicted information corresponding to the first target object is displayed on the head-up display.

2. The method of claim 1, wherein, The prediction information includes the prediction behavior corresponding to the first target object; The step of generating prediction information corresponding to the first target object based on the trajectory features and the text features includes: The trajectory features and the text features corresponding to the descriptive text are fused to generate fused features; The trajectory features, the text features, and the fused features are concatenated to generate concatenated features; Based on the splicing features, determine the probability value of the first target object performing each candidate behavior; Based on all the probability values, the predicted behavior corresponding to the first target is determined from all the candidate behaviors.

3. The method of claim 2, wherein, The text feature includes at least one text column vector; the trajectory feature includes at least one trajectory column vector; the fusion feature includes a first fusion feature and a second fusion feature; The step of fusing the trajectory features and the text features to generate fused features includes: For any text column vector, calculate the first weighting coefficient between the text column vector and each of the trajectory column vectors; The first fusion feature corresponding to the text feature is generated by weighted summation of all the text column vectors according to the first weight coefficient. For any trajectory column vector, calculate the second weighting coefficient between the trajectory column vector and each of the text column vectors; The second fusion feature corresponding to the trajectory feature is generated by weighting and summing all the trajectory column vectors according to the second weight coefficient.

4. The method of claim 3, wherein, The formula for calculating the first weighting coefficient between any text column vector and each of the trajectory column vectors is as follows: Among them, the The first weight coefficient corresponding to the i-th text column vector; the f text [i] represents the text column vector corresponding to the i-th column; the f tranditional [j] represents the trajectory column vector corresponding to the j-th column, where the value of j is in the range of [1, k1]; k1 is the total number of trajectory column vectors in the trajectory features; d1 is the preset first attention coefficient; exp represents an exponential function with the natural constant as the base. The formula for calculating the second weighting coefficient between any trajectory column vector and each of the text column vectors is as follows: Among them, the The second weight coefficient represents the trajectory column vector corresponding to the j-th column; the f text [i] represents the text column vector corresponding to the i-th column, where the value of i is in the range of [1, k2]; k2 is the total number of text column vectors in the text features; and d2 is the preset second attention coefficient.

5. The method of claim 1, wherein, The video data includes at least one video frame; the first target object includes a pedestrian; the trajectory features corresponding to the first target object include the walking trajectory features corresponding to the pedestrian. The step of inputting the video data into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object includes: For any video frame, obtain the skeletal data corresponding to each pedestrian in that video frame; For any pedestrian, generate the corresponding skeletal sequence information based on the skeletal data of the pedestrian in all video frames; Feature extraction is performed on the skeletal sequence information to generate the walking trajectory features corresponding to any pedestrian.

6. The method of claim 5, wherein, The skeletal sequence information includes joint sequence information for multiple joint points; The step of extracting features from the skeletal sequence information to generate the walking trajectory features corresponding to any pedestrian includes: Perform a first convolution operation on the joint sequence information corresponding to each joint point to determine the local motion features corresponding to each joint point. The local motion features of all joints are aggregated to determine the cooperative motion features of any pedestrian. A second convolution operation is performed on the cooperative motion features to generate the walking trajectory features corresponding to any pedestrian.

7. The method of claim 1, wherein, The video data includes at least one video frame; the first target includes at least one other vehicle traveling on the road where the driving vehicle is located; the trajectory features corresponding to the first target include the driving trajectory features corresponding to the other vehicle; The step of inputting the video data into the trajectory extraction model corresponding to the first target object to determine the trajectory features corresponding to the first target object includes: For any other vehicle, obtain the second position information of the other vehicle in each video frame of the video data; the second position information is determined after performing target detection on the video data using a first target detection algorithm; Based on the second position information of the other vehicles in any video frame, the first speed of the other vehicles in any video frame, the second speed of the driving vehicle when capturing the video frame, and the yaw rate of the driving vehicle when capturing the video frame, the trajectory data of the other vehicles in any video frame is determined. Feature extraction is performed on the trajectory data of the other vehicles in all the video frames to generate the driving trajectory features corresponding to the other vehicles.

8. The method according to any one of claims 1 to 7, characterized in that, After generating the prediction information corresponding to the first target object based on the trajectory features and the text features, the process includes: For any first target object, generate a first triplet corresponding to the first target object based on the prediction information corresponding to the first target object and the first location information corresponding to the first target object; The status information corresponding to each second target object in the video data is obtained; the second target object is a non-movable target object in the video data; the status information is determined after target detection is performed on the video data by a target detection algorithm; For any second target object, generate a second triplet corresponding to the second target object based on the state information corresponding to the second target object and the description text corresponding to the state information; Input all the first triples, all the second triples, and the preset knowledge graph into the retrieval model to determine the scene semantic graph corresponding to the video data; The scene semantic graph is input into the semantic reasoning model to generate and display the driving strategy information corresponding to the scene semantic graph.

9. An electronic device, comprising: The device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that, when the processor executes the computer program, the electronic device implements the display method as described in any one of claims 1-8.

10. A computer program product, characterised in that, Includes a computer program, which, when run, causes the display method as described in any one of claims 1-8 to be performed.

Images