Traffic scene recognition method and device

Abstract

The invention discloses a traffic scene recognition method and device. Image data and track data are obtained, image features corresponding to the image data are obtained through an image encoder, track features are obtained through a track encoder according to the track data, and text features are obtained according to context vectors and track features related to all sub-scenes of a traffic scene. And obtaining an identification result of each sub-scene of the traffic scene according to the text features and the image features. Therefore, interrelation among the sub-scenes can be established through the context vectors related to the sub-scenes of the traffic scene, the recognition accuracy of a single sub-scene is improved, meanwhile, all the sub-scenes of the traffic scene can be summarized comprehensively, and comprehensive auxiliary information is provided for downstream tasks.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a traffic scene recognition method and device. Background Art

[0002] Traffic scene recognition is a key technology in intelligent transportation systems that collects complex visual data through sensors installed on vehicles to provide meaningful and intuitive semantic information for understanding the driving environment. Traffic scene recognition helps to understand traffic conditions, concretize scene semantics, and provide auxiliary information for downstream tasks. However, current traffic scene recognition focuses on the detection of a single sub-scene, and rarely pays attention to providing a comprehensive overview of the various sub-scenes of a traffic scene. There may be relationships between the various sub-scenes, resulting in low accuracy in the recognition of a single sub-scene. Summary of the invention

[0003] In view of this, the purpose of an embodiment of the present invention is to provide a traffic scene recognition method and device, which can establish mutual connections between various sub-scenes and improve the accuracy of recognition of a single sub-scene. At the same time, it can provide a comprehensive overview of various sub-scenes of the traffic scene and provide more comprehensive auxiliary information for downstream tasks.

[0004] In a first aspect, an embodiment of the present invention provides a traffic scene recognition method, the method comprising:

[0005] Acquire collected data, wherein the collected data includes image data and trajectory data;

[0006] Acquire image features corresponding to the image data through an image encoder;

[0007] Acquiring trajectory features according to the trajectory data by means of a trajectory encoder;

[0008] Acquiring text features according to a context vector and the trajectory features, wherein the context vector is associated with each sub-scene of the traffic scene; and

[0009] Recognition results of each sub-scene of the traffic scene are obtained according to the text features and the image features.

[0010] In some embodiments, obtaining the trajectory feature according to the trajectory data by means of a trajectory encoder comprises:

[0011] Determining a target road section corresponding to the trajectory data in the road network;

[0012] Acquire a track piece corresponding to the target section according to a predetermined window, wherein the track piece includes the target section and an adjacent section, and the adjacent section is determined according to the predetermined window and the target section;

[0013] Acquire an initial hidden state according to the track sheet;

[0014] Obtaining an intermediate vector according to the initial hidden state by a position encoder; and

[0015] The trajectory feature is acquired according to the intermediate vector by the trajectory encoder.

[0016] In some embodiments, the initial hidden state includes an initial vector of each road segment in the track sheet;

[0017] The initial vector is obtained by the following steps:

[0018] Obtain static attributes and dynamic attributes of the road segment, wherein the static attributes include at least one of a road segment identification, a functional category, a number of lanes, a speed category, a road length, and a road out-degree, and the dynamic attributes include at least one of a heat map value, a planned route count, a yaw rate, and a median speed of the road segment;

[0019] generating an embedding vector according to the static attribute and the dynamic attribute; and

[0020] The initial vector is generated according to the embedding vector through a feedforward neural network.

[0021] In some embodiments, the trajectory encoder includes multiple multi-head attention layers, the input parameter of the first multi-head attention layer is the initial hidden state, the input parameters of the second and subsequent multi-head attention layers are the output parameters of the previous multi-head attention layer, and the output parameter of the last multi-head attention layer is the trajectory feature.

[0022] In some embodiments, acquiring text features according to the context vector and the trajectory features includes:

[0023] generating a first vector according to the context vector and the trajectory feature;

[0024] Generate a second vector based on the first vector through a self-attention layer;

[0025] Generate a text vector according to the second vector, the category label word and a predetermined prompt template; and

[0026] The text feature is generated according to the text vector by a text encoder.

[0027] In some embodiments, the predetermined prompt template includes a placeholder and a mask;

[0028] The step of generating a text vector according to the second vector, the category label word and the predetermined prompt template is as follows:

[0029] The second vector is added to the placeholder, and the category label word is added to the mask to generate the text vector.

[0030] In some embodiments, the method further comprises:

[0031] Obtaining the predetermined results of each sub-scenario; and

[0032] In response to the recognition result matching the predetermined result, a push message is sent to the terminal device so that the terminal device displays the recognition result.

[0033] In some embodiments, the method further comprises:

[0034] Determine the target area and target sub-scene;

[0035] Obtaining recognition results of each sub-scene of the traffic scene of each road section in the target area; and

[0036] The display information of each sub-scene of the target area is generated according to the recognition results of each sub-scene of the traffic scene of each road section.

[0037] In a second aspect, an embodiment of the present invention provides a traffic scene recognition device, the device comprising:

[0038] A data acquisition unit, used to acquire the acquired data, wherein the acquired data includes image data and trajectory data;

[0039] An image feature acquisition unit, configured to acquire image features corresponding to the image data through an image encoder;

[0040] A trajectory feature acquisition unit, used for acquiring trajectory features according to the trajectory data through a trajectory encoder;

[0041] a text feature acquisition unit, configured to acquire text features according to a context vector and the trajectory feature, wherein the context vector is related to each sub-scene of the traffic scene; and

[0042] The recognition result acquisition unit is used to acquire the recognition results of each sub-scene of the traffic scene according to the text features and the image features.

[0043] In a third aspect, an embodiment of the present invention provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0044] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing computer program instructions, wherein the computer program instructions implement the method described in the first aspect when executed by a processor.

[0045] The technical solution of the embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established to improve the recognition accuracy of a single sub-scene. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made to provide more comprehensive auxiliary information for downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0047] Figure 1 is a schematic diagram of a traffic scene recognition system according to an embodiment of the present invention;

[0048] Figure 2 is a flow chart of a traffic scene recognition method according to an embodiment of the present invention;

[0049] Figure 3 is a flow chart of obtaining trajectory features according to an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a road network according to an embodiment of the present invention;

[0051] Figure 5 is a flowchart of obtaining an initial vector according to an embodiment of the present invention;

[0052] Figure 6 is a flow chart of obtaining text features according to an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of a recognition result of a sub-scene according to an embodiment of the present invention;

[0054] Figure 8 is a schematic diagram of a data processing model of an embodiment of the present invention;

[0055] Fig. 9 is a schematic diagram of a traffic scene recognition device according to an embodiment of the present invention;

[0056] Fig.10 is a schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The present invention is described below based on embodiments, but the present invention is not limited to these embodiments. In the detailed description of the present invention below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present invention without the description of these details. In order to avoid confusing the essence of the present invention, known methods, processes, flows, components and circuits are not described in detail.

[0058] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0059] Unless the context clearly requires otherwise, the words "include", "comprising" and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning "including but not limited to".

[0060] In the description of the present invention, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0061] Figure 1 Schematic diagram of a traffic scene recognition system according to an embodiment of the present invention. Figure 1 In the illustrated embodiment, the traffic scene recognition system includes at least one vehicle 1 and a server 2 .

[0062] The vehicle 1 is equipped with a detection module, which is used to obtain collected data through the detection module and send the obtained collected data to the server 2. The collected data at least includes image data and trajectory data.

[0063] Specifically, the detection module installed on the vehicle 1 includes at least an image sensor and a positioning module. Among them, the image sensor can be implemented by a camera module, and is used to obtain image data related to the traffic scene. The positioning module is used to obtain trajectory data, and the trajectory data can be a positioning point at the current moment, or multiple positioning points within a period of time. Among them, the positioning module of the vehicle 1 can achieve positioning through various existing technologies, such as GPS (Global Positioning System) positioning, WIFI (wireless Internet) positioning, base station positioning, satellite positioning, etc. One or more combinations thereof.

[0064] The server 2 is in communication connection with the vehicle 1, and is used to generate a recognition result of the traffic scene according to the collected data after receiving the collected data. Specifically, the server obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to the text features and image features. The recognition result of the traffic scene includes the recognition results of each sub-scene of the traffic scene, the sub-scene is used to characterize different aspects of the traffic scene, and the recognition result of each sub-scene is the category of the sub-scene.

[0065] For example, the sub-scenes include one or more of road conditions, passability, lighting conditions, traffic facilities, traffic environment, road material, road width, etc. Among them, the categories corresponding to the road conditions include normal, construction, traffic accident, etc.; the categories corresponding to passability include very congested, congested, slow traffic, smooth, impassable, etc.; the categories of lighting conditions include good lighting, poor lighting, etc.; the categories of traffic facilities include traffic signs, signal lights, isolation facilities, etc.; the categories of traffic environment include store front roads, tree-lined roads, highways, urban roads, etc.; the categories of road materials include cement concrete roads, asphalt gravel roads, soil roads, brick roads, etc.; the road width includes narrow, spacious, etc.

[0066] It should be understood that the sub-scenes and sub-scene categories listed above are only examples provided by the embodiment of the present invention. The embodiment of the present invention does not limit the specific content and can be set according to actual conditions.

[0067] In some embodiments, after the server obtains the recognition result of the traffic scene, it directly sends the recognition result to the vehicle 1, so that the vehicle 1 displays the recognition result of the traffic scene through a vehicle terminal or other equipment. For the sake of convenience, the recognition results of the three sub-scenes of road condition, passability and lighting condition are obtained as an example. Assuming that the recognition result of the road condition is construction, the recognition result of passability is impassable, and the recognition result of lighting condition is good lighting, the recognition result is sent to the vehicle 1, and the vehicle 1 displays the recognition result.

[0068] In some embodiments, after the server obtains the recognition result of the traffic scene, it screens the recognition result according to the predetermined result to obtain the recognition result that meets the condition, and sends the classification result that meets the condition to the vehicle 1 for display. For the sake of convenience, the recognition results of the three sub-scenes of road condition, passability and lighting condition are obtained as an example. Assuming that the obtained recognition result of the road condition is construction, the recognition result of passability is impassable, the recognition result of the lighting condition is good lighting, and the predetermined result is "construction, impassable, poor lighting", the recognition results of the road condition and passability meet the conditions, and the road condition of construction and the passability of impassable are sent to the vehicle 1, and the vehicle 1 displays the recognition result.

[0069] In some embodiments, the traffic scene recognition system further includes a terminal device 3. Taking the online car-hailing system as an example, the terminal device 3 is a passenger terminal and / or a driver terminal. After the server 2 obtains the recognition result of the traffic scene, it sends the recognition result to the terminal device 3. The specific implementation method of the server sending the recognition result to the terminal device 3 can refer to the method of the server sending the recognition result to the vehicle, and the embodiments of the present invention will not be repeated here.

[0070] In some embodiments, the traffic scene recognition system also includes an operation and maintenance terminal 4. The operation and maintenance terminal 4 is a terminal device used by online car-hailing operation and maintenance personnel. The server determines the target area and the target sub-scene. The target area is the area where the traffic scene needs to be analyzed, and the target sub-scene is the sub-scene where the traffic scene needs to be analyzed. The target area and the target sub-scene can be set by the operation and maintenance personnel through the operation and maintenance terminal and sent to the server. For example, assuming that the road condition information of city C1 needs to be analyzed, the target area is city C1, and the target sub-scene is the road condition. The server obtains the recognition results of each sub-scene of the traffic scene of each section of the target area, collects the collected data obtained by vehicles on different sections of city C1, and obtains the recognition results of the sub-scenes of different sections according to the collected data analysis. The display information of each sub-scene of the target area is generated according to the recognition results of each sub-scene of the traffic scene of each section. The obtained display information is sent to the operation and maintenance terminal for display.

[0071] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0072] Figure 2 FIG. 1 is a flow chart of a traffic scene recognition method according to an embodiment of the present invention. Figure 2 As shown, the traffic scene recognition method of the embodiment of the present invention includes the following steps:

[0073] Step S100: Acquire collected data.

[0074] In this embodiment, obtaining the collected data specifically includes: the server receives the collected data sent by the vehicle. The collected data includes image data and trajectory data. Specifically, the detection module installed on the vehicle includes at least an image sensor and a positioning module. Among them, the image sensor can be implemented by a camera module, which is used to obtain image data related to the traffic scene. The positioning module is used to obtain trajectory data, and the trajectory data can be a positioning point at the current moment, or multiple positioning points within a period of time. Among them, the positioning module of vehicle 1 can achieve positioning through various existing technologies, such as GPS (Global Positioning System) positioning, WIFI (wireless Internet) positioning, base station positioning, satellite positioning, etc. One or more combinations thereof.

[0075] Step S200: Obtain image features corresponding to the image data through an image encoder.

[0076] In this embodiment, the principle of extracting image features through the image encoder is mainly to use the information in the image encoding process to obtain a digital representation of the image. The digital representation is the feature vector of the image feature in the embodiment of the present invention.

[0077] The image features include one or more of geometric features, shape features, amplitude features, histogram features, color features, and local binary patterns (LBP) features.

[0078] Geometric features include the position and orientation, perimeter, area, major axis and minor axis, and distance (Euclidean distance, Manhattan distance, Chebyshev distance, cosine distance, etc.) of the image;

[0079] Shape features include rectangularity, circularity, invariant moments, eccentricity, polygonal description, curve description, etc.;

[0080] Amplitude features include moments, projections, etc.;

[0081] Histogram features include mean, variance, energy, entropy, L1 norm, L2 norm, etc.

[0082] Color features include color histogram, color moment, etc.

[0083] Local binary pattern features are visual operators used for classification in the field of computer vision and are powerful features in texture classification.

[0084] The embodiment of the present invention does not limit the specific implementation method of the image encoder, which can be implemented based on various existing methods, for example, one or more combinations of a Histogram of Oriented Gradient (HOG) feature extraction algorithm, a Local Binary Pattern (LBP) feature extraction algorithm, and a Haar-like feature extraction algorithm.

[0085] Specifically, the formula for obtaining the image features corresponding to the image data through the image encoder is as follows:

[0086] T A =Encoder A (M)

[0087] Among them, T A is the image feature, Encoder A is the image encoder, and M is the image data.

[0088] Step S300: obtaining trajectory features according to the trajectory data through a trajectory encoder.

[0089] In this embodiment, the process of obtaining the trajectory features according to the trajectory data by the trajectory encoder is as follows: Figure 3 As shown, the following steps are included:

[0090] Step S310: determining a target road section corresponding to the trajectory data in the road network.

[0091] In this embodiment, the trajectory data is obtained, and the road section corresponding to the trajectory data on the road network is found by mapping coordinates.

[0092] Specifically, the road network data is acquired in advance, and the road network data includes each road section and the coordinates corresponding to the road section, and the coordinates are longitude and latitude coordinates. At the same time, as mentioned above, the trajectory data is acquired through the positioning module of the vehicle, that is, the trajectory data includes the coordinate data of the positioning point, and the coordinate data of the positioning point is matched with the coordinates of the road section to obtain the road section corresponding to the trajectory data (that is, the target road section).

[0093] Step S320: Acquire a track piece corresponding to the target road section according to a predetermined window.

[0094] In the present embodiment, since the traffic scene of a road section is more related to its adjacent road section, rather than to the road in the distance, the front and rear road section environment plays an important role in determining the traffic scene of a specific road section. Therefore, the track piece includes the target road section and the adjacent road section, and the adjacent road section is determined according to the predetermined window and the target road section. The window is used to characterize the distance between the adjacent road section and the target road section, wherein the distance is the number of road sections. For example, assuming that the size of the window is set to 1, the adjacent road section of the target road section includes the road section directly connected to the target road section. For another example, assuming that the size of the window is set to 2, the adjacent road section of the target road section includes a first road section directly connected to the target road section, and a second road section directly connected to the first road section.

[0095] Figure 4 Schematic diagram of a road network according to an embodiment of the present invention. Figure 4 In the embodiment shown, the road network in a certain area includes road segments s1-s 13 .

[0096] In an optional implementation, the adjacent road segment in the track piece is a historical driving segment and / or a road segment to be driven by the vehicle, that is, the adjacent road segment is a road segment related to the driving direction of the vehicle.

[0097] For example, assume that the vehicle's driving direction passes through sections s1, s9, s8, s4, and s5 in sequence. When the target section is s9 and the window size is 1, the adjacent sections of the target section s9 include s1 and s8. Therefore, the track piece corresponding to the target section s9 includes sections s1, s9, and s8. When the target section is s9 and the window size is 2, the first section directly connected to the target section s9 includes s1 and s8, the second section directly connected to the first section s1 is empty, and the second section directly connected to the first section s8 includes s3, s4, and s7. However, s3 and s7 are irrelevant to the vehicle's driving direction. Therefore, the track piece corresponding to the target section s9 includes sections s1, s9, s8, and s4.

[0098] In another optional implementation, the adjacent road segments in the track sheet are independent of the vehicle's driving direction.

[0099] For example, assuming that the target road segment is s9 and the size of the window is 1, the adjacent road segments of the target road segment s9 include s1 and s8. Therefore, the track piece corresponding to the target road segment s9 includes road segments s1, s9, and s8. For another example, assuming that the target road segment is s9 and the size of the window is 2, the first road segment directly connected to the target road segment s9 includes s1 and s8, the second road segment directly connected to the first road segment s1 is empty, and the second road segment directly connected to the first road segment s8 includes s3, s4, and s7. Therefore, the track piece corresponding to the target road segment s9 includes road segments s1, s9, s8, s3, s4, and s7.

[0100] Step S330: Obtain an initial hidden state according to the track sheet.

[0101] In this embodiment, the initial hidden state includes the initial vector of each road segment in the track sheet.

[0102] Assume that the track sheet includes d road segments, for the i-th (i=1, 2, ..., d) road segment s i , whose initial vector is expressed as

[0103] The process of obtaining the initial vector of each road section is as follows: Figure 5 As shown, the following steps are included:

[0104] Step S331: Obtain static attributes and dynamic attributes of the road section.

[0105] In this embodiment, the static attributes include at least one of a road segment identification, a functional category, a number of lanes, a speed category, a road length, and a road out-degree, and the dynamic attributes include at least one of a heat map value of the road segment, a planned route count, a yaw rate, and a median speed.

[0106] Road segments are usually associated with various contextual features, including static and dynamic attributes. Given a road segment s i , the context feature of the road segment itself is represented as Embedding vector for static features It includes six road attributes for context embedding, namely the embedding vector v of the road segment ID , the embedding vector v of the functional category FC , the embedding vector v of the number of lanes LN , the embedding vector v of the speed category SC , the embedding vector v of the road length RL and the embedding vector v of the road out-degree OD In addition, the historical road trajectories over a period of time (e.g., one month, etc.) are analyzed and the embedding vectors of dynamic features are calculated. Including through sections i The embedding vector v of the heat map value HV , the embedding vector v of the planned route count RC , the embedding vector v of the yaw rate DR and the median velocity embedding vector v MS .

[0107] Step S332: Generate an embedding vector according to the static attribute and the dynamic attribute.

[0108] In this embodiment, for continuous features, the entire value range is dispersed into several continuous small intervals, and the interval number is used for feature encoding. In this way, a unique embedding is set for each discrete value or interval number, and then the related feature vectors are connected as the road segment s i The complete context is embedded to generate embedding vectors for static attributes and the dynamic attributes.

[0109] The embedding vector of static attributes is calculated as:

[0110]

[0111] in, For road segment s i The embedding vector of the static attributes, v ID is the embedding vector of the road segment, v FC is the embedding vector of the functional category, v LN is the embedding vector of the number of lanes, v SC is the embedding vector of the speed category, v RL is the embedding vector of road length, v OD is the embedding vector of the road out-degree, and || represents the vector concatenation operation.

[0112] The embedding vector of dynamic attributes is calculated as:

[0113]

[0114] in, For road segment s i The embedding vector of the dynamic attributes, v HV is the embedding vector of the heat map value, v RC is the embedding vector of the planned route count, v DR The embedding vector of the yaw rate, v MS is the embedding vector of the median velocity.

[0115] Embedding vector of static attributes and the embedding vector of the dynamic attributes Concatenate to get the embedding vector of the road segment The specific formula is as follows:

[0116]

[0117] in, For road segment s i The embedding vector of the static attributes of For road segment s i The embedding vector of the dynamic attributes of For road segment s i The embedding vector of , || represents the vector concatenation operation.

[0118] It should be noted that the embedding vector listed above is only an example provided by the embodiment of the present invention. The embodiment of the present invention can flexibly configure the context features according to the actual application scenario.

[0119] Step S333: Generate the initial vector according to the embedded vector through a feedforward neural network.

[0120] In this embodiment, a feedforward neural network is used to convert the embedding vector into an initial vector. The specific formula is as follows:

[0121]

[0122] in It is road section i The initial vector, For road segment s i FFN(·) represents a two-layer fully connected network, i.e., a feed-forward neural network.

[0123] Furthermore, the above method is repeated to obtain the initial vector of each road segment in the track piece, and then the initial hidden state of the track piece is generated.

[0124] For the track sheet p i =[s1,s2,……,s d ], its initial hidden state can be expressed as d represents the number of road segments in a track patch.

[0125] Step S340: Obtain an intermediate vector according to the initial hidden state through a position encoder.

[0126] In this embodiment, in order to enhance the attention mechanism to capture the order of the road segments in the track sheet, the position information of the road segments is added to the initial hidden state through the position encoding layer to ensure that the attention mechanism does not ignore the timing information. The specific calculation method is:

[0127]

[0128] in, is the initial hidden state, and PE represents the location information of the road segment obtained by the position encoding layer.

[0129] Among them, the position encoding layer can represent the position information of each initial vector in the initial hidden state.

[0130] Step S350: acquiring the trajectory feature according to the intermediate vector through the trajectory encoder.

[0131] In this embodiment, the trajectory encoder includes multiple multi-head attention layers, the input parameter of the first multi-head attention layer is the initial hidden state, the input parameters of the second and subsequent multi-head attention layers are the output parameters of the previous multi-head attention layer, and the output parameter of the last multi-head attention layer is the trajectory feature.

[0132] Furthermore, each multi-head attention layer includes an input mapping layer, a scaled dot product attention layer, a merge layer, and an output mapping layer.

[0133] Assuming that the trajectory encoder includes N multi-head attention layers, for the l-th multi-head attention layer, the input parameters are expressed as:

[0134] Q (l) =K (l) =V (l) =H (l-1)

[0135] Among them, Q (l) is the Q(query) value of the l-th multi-head attention layer, K (l) is the K(key) value of the l-th multi-head attention layer, V (l) is the V(value, value) value of the l-th multi-head attention layer, H (l-1) is the hidden state input to the l-th multi-head attention layer.

[0136] It should be noted that in the multi-head attention mechanism, the Q value is used for query, indicating the object to be paid attention to, the K value is used to calculate the correlation between the query and the key, and the V value stores the information content to be queried. At the same time, when l = 1, H (0) is the intermediate vector obtained in step S340.

[0137] Specifically, the multi-head attention layer is used to share the features of different sections in the trajectory. The specific calculation formula is as follows:

[0138]

[0139] Among them, Concat represents the concatenation operation, W O is the learnable parameter matrix, W O Obtained through pre-training, is the first subvector output by the input mapping layer, h is the number of heads in the multi-head attention layer, head i Represents the output of the i-th head in the multi-head attention layer, i = 1, 2, ..., h.

[0140] Furthermore, the output head of the i-th head in the multi-head attention layer i The calculation formula is as follows:

[0141]

[0142] in, and is the learnable parameter matrix.

[0143] Among them, the above calculation head i In the formula, Attention() is implemented as follows:

[0144]

[0145] Among them, d is the dimension of the latent space, K T represents the transposed matrix of K, and softmax() represents the normalization function.

[0146] Then another feed-forward neural network layer is used to learn the features Transform and use a residual connection and layer normalization to generate the output of the lth module:

[0147]

[0148] in, Represents a feedforward neural network layer pair Transform, LayerNorm() represents the normalization function, H (l) is the output of the l-th multi-head attention layer.

[0149] Furthermore, the input parameter of the first multi-head attention layer is the initial hidden state, the input parameter of the second and subsequent multi-head attention layers is the output parameter of the previous multi-head attention layer, and the output parameter H of the last multi-head attention layer is (N) is the trajectory feature. That is:

[0150] R=H (N)

[0151] Among them, R is the trajectory feature, H (N) is the output of the last multi-head attention layer.

[0152] Step S400: Acquire text features according to the context vector and the trajectory features.

[0153] In this embodiment, a context vector is obtained through pre-training, and the context vector is related to each sub-scenario of the traffic scene.

[0154] The process of obtaining text features based on context vectors and trajectory features is as follows: Figure 6 As shown, the specific steps include:

[0155] Step S410: Generate a first vector according to the context vector and the trajectory feature.

[0156] In this embodiment, the context vector C is m*n c A vector of, where m represents the number of sub-scenes, n c is the number of context words, where the dimension of each word is d.

[0157] In order to utilize traffic domain knowledge to enhance the construction of learnable prompt templates, the context vector C and trajectory features R are simultaneously incorporated into the model to obtain the first vector:

[0158]

[0159] Among them, C is the context vector, R is the trajectory feature, is the first vector, Represents an element-wise addition operation on a matrix.

[0160] Step S420: Generate a second vector based on the first vector through a self-attention layer.

[0161] In order to promote knowledge sharing among different sub-scenes, a self-attention mechanism is used to capture the interdependencies among different sub-scenes. The calculation formula for obtaining the second vector is as follows:

[0162]

[0163] Among them, SelfAttention() represents the self-attention layer algorithm, is the first vector, is the second vector.

[0164] Among them, the SelfAttention layer is modeled as follows:

[0165]

[0166]

[0167]

[0168] in, The second vector The i-th column vector of The first vector The i-th column vector of The first vector The kth column vector of express and The attention between them, k′ is the first vector The total number of column vectors (equal to n c ), w1 and w2 are learnable weight parameters, exp(·) represents the exponential function, and tanh(·) represents the hyperbolic tangent function.

[0169] In this way, linking the features of different sub-scenarios helps to gain a comprehensive understanding of traffic scenarios.

[0170] Step S430: Generate a text vector according to the second vector, the category label word and a predetermined prompt template.

[0171] In this embodiment, the preset prompt template is:

[0172]

[0173] Among them, t i Represents a prompt template, is a placeholder, [Mask i ] is a mask character.

[0174] That is, the predetermined prompt template includes a placeholder and a mask, and the length of the placeholder is the same as the length of the row vector of the second vector, both of which are n c .

[0175] Furthermore, generating the text vector according to the second vector, the category label word and the predetermined prompt template is specifically: adding the second vector to the placeholder, adding the category label word to the mask to generate the text vector.

[0176] In this embodiment, the second vector obtained by step S420 is m*n c , add each row vector to the placeholder in the prompt template, and add the category label word to the mask to obtain the text vector corresponding to each row vector.

[0177] That is, the second vector Divide by row vector to get m contexts, and record the i-th context as Finally, t i The embedded features of the placeholders in are replaced by Then replace the mask with the category label word to complete the construction of a learnable text representation Represents the text vector when the category label of the i-th sub-scene is j.

[0178] For example, for the second vector use Represents the second vector The i-th row of . Assume that the i-th sub-scene includes two categories, that is, j = 1 or 2, and the category labels corresponding to the two categories are Y1 and Y2 respectively. Then for the i-th sub-scene, there are two text vectors generated, which are:

[0179]

[0180]

[0181] In this way, the text vector corresponding to each sub-scene can be obtained, and the number of text vectors corresponding to each sub-scene is the same as the number of its category labels.

[0182] Step S440: Generate the text feature according to the text vector through a text encoder.

[0183] In this embodiment, the text vector is input into a pre-trained text encoder to obtain text features. The specific calculation formula is as follows:

[0184]

[0185] in, represents the text feature when the category label of the i-th sub-scene is j, Represents the text vector when the category label of the i-th sub-scene is j. t Represents a text encoder.

[0186] It should be understood that the embodiments of the present invention do not limit the implementation method of the text encoder, which can be implemented based on various existing methods.

[0187] In some embodiments, each string in the text feature is converted to a lowercase byte pair encoding (BPE) representation, which assigns a unique digital ID to each string. In order to achieve efficient batch processing, each text sequence is padded with a special sequence start string ([SOS]) and sequence end string ([EOS]) to fix its length to 77 strings.

[0188] Step S500: Obtain recognition results of each sub-scene of the traffic scene according to the text features and the image features.

[0189] In this embodiment, as described above, through steps S100 - S400 , text features of each sub-scene when it is of various categories can be obtained, and the text features are compared with the image features to obtain the recognition results of each sub-scene.

[0190] In some embodiments, the text features and the image features are both vectors, and the similarity between each text feature and the image feature is calculated for each sub-scene corresponding to the multiple text features, and the category label word corresponding to the text feature with the highest similarity to the image feature is obtained, and the category corresponding to the category label word is determined as the recognition result of the sub-scene. Repeating the execution can obtain the recognition result of each sub-scene.

[0191] The similarity between the text feature and the image feature can be obtained by various existing methods, such as Euclidean distance, Manhattan distance or cosine similarity.

[0192] In a specific implementation, the formula for calculating the similarity between image features and text features is as follows:

[0193]

[0194] Where t is the temperature parameter, is the similarity, T A is the image feature, is a text feature.

[0195] The embodiment of the present invention integrates traffic domain knowledge into the pre-trained visual language model, uses road network attributes and historical vehicle trajectories, extracts traffic features through a special trajectory encoder, and fuses the extracted traffic features with the original learnable prompt template. The self-attention mechanism between the learnable prompt templates establishes connections between different sub-scenes, promotes the mutual connection between different sub-scenes of the traffic scene, and thus generates a more comprehensive and consistent description.

[0196] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0197] In some embodiments, the method further comprises:

[0198] Step S610: Obtain the predetermined result of each sub-scenario.

[0199] Step S620: In response to the recognition result matching the predetermined result, sending a push message to the terminal device so that the terminal device displays the recognition result.

[0200] For steps S610-S620, after the server obtains the recognition results of the traffic scene, it filters the recognition results according to the predetermined results to obtain the recognition results that meet the conditions, and sends the classification results that meet the conditions to the terminal device for display. For the sake of convenience, the recognition results of the three sub-scenes of road condition, passability and lighting conditions are obtained as an example. Assume that the obtained recognition result of the road condition is construction, the recognition result of passability is impassable, and the recognition result of the lighting condition is good lighting. The predetermined result is "construction, impassable, poor lighting", then the recognition results of the road condition and passability meet the conditions, and the road condition is construction and the passability is impassable. It is sent to the terminal device, and the terminal device displays the recognition results. Among them, the terminal device can be a driver terminal, a passenger terminal or a vehicle-mounted terminal, etc.

[0201] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0202] In some embodiments, the method further comprises:

[0203] Step S710: determine the target area and target sub-scene.

[0204] Step S720: Obtain recognition results of each sub-scene of the traffic scene of each road section in the target area.

[0205] Step S730: Generate display information of each sub-scene of the target area according to the recognition results of each sub-scene of the traffic scene of each road section.

[0206] For steps S710-S730, the server determines the target area and the target sub-scene. The target area is the area where the traffic scene needs to be analyzed, and the target sub-scene is the sub-scene of the traffic scene that needs to be analyzed. The target area and the target sub-scene can be set by the operation and maintenance personnel through the operation and maintenance terminal and sent to the server. For example, assuming that the road condition information of city C1 needs to be analyzed, the target area is city C1, and the target sub-scene is the road condition. The server obtains the recognition results of each sub-scene of the traffic scene of each section of the target area, collects the collected data obtained by vehicles on different sections of city C1, and obtains the recognition results of the sub-scenes of different sections based on the collected data analysis. The display information of each sub-scene of the target area is generated based on the recognition results of each sub-scene of the traffic scene of each section. The obtained display information is sent to the operation and maintenance terminal for display. The operation and maintenance terminal is a terminal device used by the online car-hailing operation and maintenance personnel.

[0207] Figure 7 Schematic diagram of the recognition results of sub-scenes according to an embodiment of the present invention. For a certain area, after the recognition results of each sub-scene are obtained, points rendered in different ways (such as color rendering, etc.) represent different categories of each sub-scene.

[0208] The recognition results of the four sub-scenes obtained by the prior art are as follows: Figure 7 As shown in G1, G2, G3, and G4 in FIG, the recognition results of the four sub-scenes obtained by the traffic scene recognition method of the embodiment of the present invention are respectively as follows: Figure 7 As shown in G5, G6, G7, and G8, G1 and G5 are the same sub-scene, G2 and G6 are the same sub-scene, G3 and G7 are the same sub-scene, and G4 and G8 are the same sub-scene.

[0209] As can be seen from the figure, there is a clear overlap between different categories in the prior art. In addition to the different clusters that can be observed in the scene data set, the different categories in other data sets are mixed with each other, without obvious separation, and the recognition effect is poor. In the recognition results of the embodiment of the invention, there are clear boundaries between different categories of sub-scenes. In addition, although there are significant differences in the number of samples of different categories in some tasks, the model still captures the characteristics of each category.

[0210] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0211] The above is the process of obtaining the recognition results of each sub-scene based on the collected data through the pre-trained model. The model training and recognition processes are similar, specifically:

[0212] Acquire training samples, wherein the training samples include acquisition data and sample labels, wherein the sample labels include the categories of each sub-scene, and the acquisition data includes image data and trajectory data. At the same time, obtain an initial context vector and an initial trajectory encoder, obtain the recognition results of each sub-scene under the conditions of the initial context vector and the initial trajectory encoder through a process similar to the above, and update the initial context vector and the initial trajectory encoder according to the recognition results and the sample labels. Repeat the above process to obtain a pre-trained context vector and trajectory encoder.

[0213] It should be noted that the text encoder and trajectory encoder of the embodiment of the present invention can both adopt the text encoder and trajectory encoder with trained frozen parameters. Therefore, during the training process, it is not necessary to update the parameters of the text encoder and trajectory encoder.

[0214] In a specific implementation, a multi-task learning strategy is adopted, and the loss function is defined as the sum of cross entropies in different aspects:

[0215]

[0216]

[0217] Among them, Loss is the loss value, represents the label of the jth category in sub-scene i, N i represents the number of classifications of sub-scene i, and m is the number of sub-scenes.

[0218] The context vector and trajectory encoder parameters are updated according to the loss value until the training is completed when the predetermined conditions are met.

[0219] After the training process is completed, the traffic scene images and corresponding vehicle trajectories are input into the model to obtain the most likely classification results for different sub-scenes. These results are then filled into the mask field of the predetermined template to obtain a comprehensive description of the traffic scene.

[0220] Figure 8 Schematic diagram of a data processing model according to an embodiment of the present invention. Figure 7 In the illustrated embodiment, the data processing model includes an image encoder 71, a road network mapping layer 72, an embedding layer 73, a position encoder 74, a first merging layer 75, a trajectory encoder 76, a second merging layer 77, a self-attention layer 78, a prompt template 79, and a text encoder 80. Figure 2-Figure 6 The specific implementation of each functional module of the traffic scene recognition method and data processing model is as follows:

[0221] The image encoder 71 is used to obtain image features according to the image data, as shown in step S200.

[0222] The road network mapping layer 72 is used to determine the target road segment corresponding to the track data in the road network, and obtain the track piece corresponding to the target road segment according to a predetermined window. For details, please refer to steps S310-S320.

[0223] The embedding layer 73 is used to obtain an initial hidden state according to the track sheet. For details, please refer to step S330.

[0224] The position encoder 74 is used to add the position information of the road segment in the initial hidden state to ensure that the attention mechanism does not ignore the timing information.

[0225] The first merging layer 75 is used to merge the position information with the initial hidden state to generate an intermediate vector. For details, please refer to step S340.

[0226] The trajectory encoder 76 is used to obtain the trajectory feature according to the intermediate vector. For details, please refer to step S350.

[0227] The second merging layer 77 is used to obtain the first vector according to the context vector and the trajectory feature. For details, please refer to step S410.

[0228] The self-attention layer 78 is used to generate a second vector based on the first vector. For details, please refer to step S420.

[0229] The prompt template 79 is used to generate a text vector according to the second vector and the category label word. For details, please refer to step S430.

[0230] The text encoder 80 is used to generate the text feature according to the text vector. For details, please refer to step S440.

[0231] Afterwards, the recognition results of each sub-scene of the traffic scene are obtained according to the text features and the image features.

[0232] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0233] Fig. 9 Schematic diagram of a traffic scene recognition device according to an embodiment of the present invention. Fig. 9 In the illustrated embodiment, the traffic scene recognition device includes a collection data acquisition unit 91, an image feature acquisition unit 92, a trajectory feature acquisition unit 93, a text feature acquisition unit 94, and a recognition result acquisition unit 95. The collection data acquisition unit 91 is used to acquire collection data, and the collection data includes image data and trajectory data. The image feature acquisition unit 92 is used to acquire image features corresponding to the image data through an image encoder. The trajectory feature acquisition unit 93 is used to acquire trajectory features according to the trajectory data through a trajectory encoder. The text feature acquisition unit 94 is used to acquire text features according to a context vector and the trajectory features, wherein the context vector is related to each sub-scene of the traffic scene. The recognition result acquisition unit 95 is used to acquire recognition results of each sub-scene of the traffic scene according to the text features and the image features.

[0234] In some embodiments, the trajectory feature acquisition unit includes:

[0235] a target road section determination subunit, used to determine a target road section corresponding to the trajectory data in a road network;

[0236] A track piece acquisition subunit, configured to acquire a track piece corresponding to the target section according to a predetermined window, wherein the track piece includes the target section and an adjacent section, and the adjacent section is determined according to the predetermined window and the target section;

[0237] An initial hidden state acquisition subunit, used for acquiring an initial hidden state according to the track sheet;

[0238] an intermediate vector acquisition subunit, configured to acquire an intermediate vector according to the initial hidden state through a position encoder; and

[0239] The trajectory feature acquisition subunit is used to acquire the trajectory feature according to the intermediate vector through the trajectory encoder.

[0240] In some embodiments, the initial hidden state includes an initial vector of each road segment in the track sheet;

[0241] The initial vector is obtained by the following steps:

[0242] Obtain static attributes and dynamic attributes of the road segment, wherein the static attributes include at least one of a road segment identification, a functional category, a number of lanes, a speed category, a road length, and a road out-degree, and the dynamic attributes include at least one of a heat map value, a planned route count, a yaw rate, and a median speed of the road segment;

[0243] generating an embedding vector according to the static attribute and the dynamic attribute; and

[0244] The initial vector is generated according to the embedding vector through a feedforward neural network.

[0245] In some embodiments, the trajectory encoder includes multiple multi-head attention layers, the input parameter of the first multi-head attention layer is the initial hidden state, the input parameters of the second and subsequent multi-head attention layers are the output parameters of the previous multi-head attention layer, and the output parameter of the last multi-head attention layer is the trajectory feature.

[0246] In some embodiments, the text feature acquisition unit includes:

[0247] A first vector generating subunit, configured to generate a first vector according to the context vector and the trajectory feature;

[0248] A second vector generating subunit, configured to generate a second vector according to the first vector through a self-attention layer;

[0249] a text vector generating subunit, configured to generate a text vector according to the second vector, a category label word and a predetermined prompt template; and

[0250] The text feature generating subunit is used to generate the text feature according to the text vector through a text encoder.

[0251] In some embodiments, the predetermined prompt template includes a placeholder and a mask;

[0252] Wherein, the text vector generating subunit is specifically used for:

[0253] The second vector is added to the placeholder, and the category label word is added to the mask to generate the text vector.

[0254] In some embodiments, the apparatus further comprises:

[0255] A predetermined result obtaining unit, used to obtain a predetermined result of each sub-scenario; and

[0256] The push information sending unit is used to send push information to the terminal device in response to the recognition result matching the predetermined result, so that the terminal device displays the recognition result.

[0257] In some embodiments, the apparatus further comprises:

[0258] A target information determination unit, used to determine a target area and a target sub-scene;

[0259] an identification unit, configured to obtain identification results of each sub-scene of the traffic scene of each road section in the target area; and

[0260] The display information generating unit is used to generate the display information of each sub-scene of the target area according to the recognition results of each sub-scene of the traffic scene of each road section.

[0261] The embodiment of the present invention obtains image data and trajectory data, obtains image features corresponding to the image data through an image encoder, obtains trajectory features according to the trajectory data through a trajectory encoder, obtains text features according to context vectors and trajectory features related to each sub-scene of the traffic scene, and obtains recognition results of each sub-scene of the traffic scene according to text features and image features. Thus, through the context vectors related to each sub-scene of the traffic scene, the mutual connection between each sub-scene can be established, and the accuracy of the recognition of a single sub-scene can be improved. At the same time, a comprehensive overview of each sub-scene of the traffic scene can be made, providing more comprehensive auxiliary information for downstream tasks.

[0262] Fig.10 is a schematic diagram of an electronic device according to an embodiment of the present invention. Fig.10The electronic device shown is a general data processing device, which includes a general computer hardware structure, which at least includes a processor 101 and a memory 102. The processor 101 and the memory 102 are connected via a bus 103. The memory 102 is suitable for storing instructions or programs executable by the processor 101. The processor 101 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 101 executes the instructions stored in the memory 102, thereby executing the method flow of the embodiment of the present invention as described above to realize the processing of data and the control of other devices. The bus 103 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 104 and the display device and the input / output (I / O) device 105. The input / output (I / O) device 105 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 105 is connected to the system through an input / output (I / O) controller 106.

[0263] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, devices (equipment) or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0264] The present invention is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0265] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0266] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0267] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0268] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A traffic scene recognition method, characterized in that: The method comprises: Acquire collected data, wherein the collected data includes image data and trajectory data; Acquire image features corresponding to the image data through an image encoder; Acquiring trajectory features according to the trajectory data by means of a trajectory encoder; Acquiring text features according to a context vector and the trajectory features, wherein the context vector is associated with each sub-scene of the traffic scene; and Recognition results of each sub-scene of the traffic scene are obtained according to the text features and the image features.

2. The method according to claim 1, characterized in that The obtaining of the trajectory feature according to the trajectory data by means of the trajectory encoder comprises: Determining a target road section corresponding to the trajectory data in the road network; Acquire a track piece corresponding to the target section according to a predetermined window, wherein the track piece includes the target section and an adjacent section, and the adjacent section is determined according to the predetermined window and the target section; Acquire an initial hidden state according to the track sheet; Obtaining an intermediate vector according to the initial hidden state by a position encoder; and The trajectory feature is acquired according to the intermediate vector by the trajectory encoder.

3. The method according to claim 2, characterized in that The initial hidden state includes an initial vector of each road segment in the track sheet; The initial vector is obtained by the following steps: Obtain static attributes and dynamic attributes of the road segment, wherein the static attributes include at least one of a road segment identification, a functional category, a number of lanes, a speed category, a road length, and a road out-degree, and the dynamic attributes include at least one of a heat map value, a planned route count, a yaw rate, and a median speed of the road segment; generating an embedding vector according to the static attribute and the dynamic attribute; and The initial vector is generated according to the embedding vector through a feedforward neural network.

4. The method according to claim 2, characterized in that: The trajectory encoder includes multiple multi-head attention layers, the input parameter of the first multi-head attention layer is the initial hidden state, the input parameters of the second and subsequent multi-head attention layers are the output parameters of the previous multi-head attention layer, and the output parameter of the last multi-head attention layer is the trajectory feature.

5. The method according to claim 2, characterized in that: The acquiring of text features according to the context vector and the trajectory features comprises: generating a first vector according to the context vector and the trajectory feature; Generate a second vector based on the first vector through a self-attention layer; Generate a text vector according to the second vector, the category label word and a predetermined prompt template; and The text feature is generated according to the text vector by a text encoder.

6. The method according to claim 5, characterized in that The predetermined prompt template includes a placeholder and a mask; The step of generating a text vector according to the second vector, the category label word and the predetermined prompt template is as follows: The second vector is added to the placeholder, and the category label word is added to the mask to generate the text vector.

7. The method according to claim 1, characterized in that The method further comprises: Obtaining the predetermined results of each sub-scenario; and In response to the recognition result matching the predetermined result, a push message is sent to the terminal device so that the terminal device displays the recognition result.

8. The method according to claim 1, characterized in that The method further comprises: Determine the target area and target sub-scene; Obtaining recognition results of each sub-scene of the traffic scene of each road section in the target area; and The display information of each sub-scene of the target area is generated according to the recognition results of each sub-scene of the traffic scene of each road section.

9. A traffic scene recognition device, characterized in that: The device comprises: A data acquisition unit, used to acquire the acquired data, wherein the acquired data includes image data and trajectory data; An image feature acquisition unit, configured to acquire image features corresponding to the image data through an image encoder; A trajectory feature acquisition unit, used for acquiring trajectory features according to the trajectory data through a trajectory encoder; a text feature acquisition unit, configured to acquire text features according to a context vector and the trajectory feature, wherein the context vector is related to each sub-scene of the traffic scene; and The recognition result acquisition unit is used to acquire the recognition results of each sub-scene of the traffic scene according to the text features and the image features.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium storing computer program instructions, characterized in that: The computer program instructions implement the method according to any one of claims 1 to 8 when executed by a processor.

Images